TurnKey Linux Virtual Appliance Library

TKLPatch for developing patches (idea)

Hi guys! I was thinking in releasing a TKLPatch for TKLPatch Development. :) It consists of Turnkey Core+tklpatch installed+Turnkey Core image inside by default. Maybe son helper scripts to create the basic tklpatch structure and the helper functions (install, cleanup_apt).

I also thought of using apt-cacher (mentioned by Jed in other thread) to create a "deb cache" for the appliances, so that if you have a slow connection, once you've downloaded apache for the first patch, the next ones will use the cached deb. The only thing that stops me on this is telling tklpatch-chroot to change source apt repos to localhost and set the originals on exit. It may be a function for conf, but then it won't be a default behaviour. 

Also, a shared folder for output (or an ftp server by default) will leave your isos ready to get them (I usually scp them but to make it easier for newcomers to take the results "out of there!")

What do you think of this idea? This would help other people to apply TKLPatchs as the only need to install the Dev Env appliance. And the apt-cache will help developers speed up their work. 

I'll work on it when I finish the ones I'm making, in the meantime feedback is welcome. 

P.S: Looking into the feature you can even have a webmin module to apply patches :D maybe it reads an online approved official TKLPatchs lists with info about the base used (if not turnkey core) and automatically donwload the base (to the turnkey base images folder) and apply the patch, leaving it in the patched iso's folder (with a nice name set in the info file too, so that you don't end up with tkl-patched file).

Jeremy's picture

Interesting idea Adrian

I must admit I quite like it. It'd be sort of like a TKL Dev Workbench appliance! The idea of having apt-cacher integrated would be useful for any TKL dev work I reckon. Even if you have a fast connection with unlimited downloads, its still tons faster to install a deb that is locally cached rather than have to download it. It also means less bandwidth and load on the servers mirrorring the packages which has to be worth some OSS brownie points!?

But as you suggest, making apt-cacher work nicely in the patching process (without leaving broken ISOs) may be tricky. I'm fresh out of ideas how to do it!

Liraz Siri's picture

Great idea I never considered for some reason

A value-added TKLPatch development environment could be pretty neat, especially if you can get the integration with apt-cacher to work seamlessly!

Please feel welcome to hack TKLPatch itself if you need to do that to enable certain functionality. We'll definitely consider including those patches in the next version of TKLPatch so long as they don't regress the usability of the default use case.

Regarding future features (e.g., webmin module for applying patches), I would recommend we keep it simple for now and then think long and hard about anything that would require writing substantial amounts of new code. If you don't design things right you're going to get things wrong and have to throw most of it away eventually. Measure twice, cut once.

Jeremy's picture

Ok if we're hacking TKLPatch

but without affecting its normal functionality, here's my idea. How about getting TKLPatch to check to see if apt-cacher is installed on the host system, if so then call a script to work out the appropriate 01proxy config (ie IP address) and add it prior to applying the patch and remove it afterwards?

Not sure if that's the best way but my 2c :)

Am I being given a licence to hack? YEAH!!!!!

I've been thinking in the easiest and of course functional solution to this issue of using apt-cacher. As I have to change the repo's url in the files, this has to be done with care, so that the final system is built with the correct urls. The case is more complex if you use external repos. You should be able to cache them too. But I'll work first on a solution for ubuntu's official repos. 

Yes, I'll check TKLPatch code and try to do some hacking in a way that even Alon won't know that it's have been hacked :) So don't tell him anything please. 

Jeremy's picture

I think you missed the 01proxy option to set up apt-cacher

You do not need to touch urls, source.list or any of that. All you need to do is add a /etc/apt/apt.conf.d/01proxy file with details of your apt-cacher machine (in the this case it'll be the IP of the base TKLPatch appliance). For example this would be the contents of the 01proxy file using apt-cacher-ng defaults when the IP is xxx.xxx.xxx.xxx:

Acquire::http { Proxy “http://xxx.xxx.xxx.xxx:3142″; };

It could either exist somewhere already and be copied in or be created on the fly. It can then be simply removed with rm when done.

This will still allow the chroot environment to still use non standard repos that the host machine doesn't have. Apt-cacher-ng keeps a track of all repos it is using.

Apt-cacher-ng is the prefered option (rather than the older more RAM intensive apt-cacher). It also has a nice web interface that can be found by browising to http://xxx.xxx.xxx.xxx:3142/acng-report.html

WOW thanks!

I haven't look at the docs yet as I'm finishing the other appliances. Thanks Jed, this will make me go straight to what I'm looking for. And it resolves the main issue. It's going to be real simple like this. 

Nice! :)

I'll be using the apt-cacher-ng option for this appliance

as it makes more sense in this particular scenario. But I've just saw this and it might be a candidate for an appliance: 

 

http://www.linuxjournal.com/content/presenting-squid-deb-proxy-speed-your-update-downloads

I think it's very useful if you have a big network to have a deb cache in place. I'm going to setup one for home! 

YOHOOOOOOOOOOOOOOOOOOOO!!!!

Blazing fast package downloading!!! :D

Sorry, I just got exited with my TKLPatch for TKLPatch Dev prototype :) 

Do you people think that adding a squid to the mix could be of benefit? For those patches that download tar files... 

I made a small surgery to tklpatch-chroot to add and remove the proxy reference. So far is very transparent for the final user. Maybe the creators itself don't find it mwuahahahahaha. 

BTW

I'm still looking for a good way to clean the environment. Liraz suggested in an old thread that fuser -K could resolve this issue, but in my tests, I've been unsuccessful trying to release-and-delete the .rootfs directory after an explosion in the TKL Lab... So more ideas are welcome. I plan to make an extra command to clean the environment. I also was thinking on adding a parameter to tkl commands to look files on specific places if they are not present in the current directory. That way the appliance will have a turnkey-image-base dir (under /var/tklpatch/images?) with turnkey core by default. And an output dir for the process (the shared folder) by default. Say /srv/tklpatch/patched-isos/? Or better! a command to "publish" the result, (that internally copies it to the shared folder). 

MMMM some feedback please! Thanks... 

Jeremy's picture

Sounds like you're having fun :)

I think a simple squid proxy for tarballs is a great idea, not sure how that would interact with apt-cacher-ng though? I'm pretty sure I've read a bit about that somewhere (some tweaking req'd perhaps?) but can't recall where sorry (Ubuntu forums I suspect).

It sounds like your TKLPatch hacking is going nicely!

As for your cleanup issues, sorry no clever ideas here.

I really like your idea of having a default place for base and patched images, that makes lots of sense. This may be overkill but what about also having it as a Samba share or http access to these for Win users (I suspect that many TKL users use Win desktops)? Just an idea - and probably over complicating things but may be a nice added extra?

It's not overkill

From the beginning I was thinking of a samba share, with the same justification you are explaining, windows users. So yes, that's in the plan. I've been delaying things for the TKLCore Client to work on this one, so that I can test it enough making the other patches.

Thanks for the feedback!

flexbean's picture

Awesome idea Adrian

First things first, kudo's and cheers to this TKL team and it's leadership. I've been looking at JumpBox, Bitnami and TKL and I've been impressed with the tech and your warm community spirit. Keep up the awesome work gentlemen!

I was thinking how nice it would be to install TKLPatches from WebMin as a module and then I saw this thread. Some advantages I'm seeing:

1) Users can copy local VM images and only download TKLPatches

2) Lower the barrier to entry for testers of TKLPatches as they can just go to WebMin and try things out

3) Assist users in building combination TKL deployments similar to Bitnami stacks.

Thoughts?


Thanks flexbean

I must tell you that I'm actively working in this project and you can see the advance here. Some of my thoughts on this:

- A Webmin module didn't seem right, as Webmin is mostly a system administration frontend, while the whole tklpatch subject is a "product". So I decided to make this a standalone web application instead. I already started development and this weekend I'll be posting a screencast with the preview. So keep watching the thread I mentioned earlier. 

- You got the main idea: lower the barrier for testers AND developers of TKLPatches. It's interesting how this appliance is perfectly targeted for beginners and advanced users alike. For beginners: easy patching for testing, easy development of new patches. For advanced users/developers: powerful tools and commands to make your work fast. An easy way to help beginners into testing the patches. 

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)