TKLPatch for TKLPatch Development Environment v0.1 [Updated 07/11/2010]

This is a great idea. The only thing I thought as reading you beat me to! (Under 'future ideas') a share for the patches themselves (to load and or to get out once developed). Or maybe they could go in the other folders with the isos?

Regardless, good work and I'll have to have a play with this sometime real soon.

The thing which I like most in this patch is caching provided by squid-proxy ..... will it work for svn,git and mercurial repos ? Also is there any way by which caching can be made in gem install commands ?

Regarding the name , I prefer  "TKL-ADE: TurnkeyLinux Appliance Development Environment"

How about adding zsync to update Core image to latest one by downloading the changed bits only [http://www.webupd8.org/2009/10/how-to-upgrade-your-karmic-iso-with.html] . I know it requires a zsync server on turnkeylinux website.

I found change the hostname, but why not use FQDN? some program like mail would need use .

change the hostname on installation. Let's hope so, as it is necesary to be able to customize the FQDN for many appliances. Also regional settings would be nice. 

This sounds so cool Adrian. I'm excited about the possibilities! You've obviously put a lot of thought into this! These ideas will make community involvement so much easier.

I think that is a really good thing. Having this available as an appliance will  potentially allow for some niche community interests to be accommodated without requiring dev's approval and/or support. Another advantage of making patching more accessable, is that hopefully it will get more people involved in patching and development. Often all it takes is some small successes to buoy people's enthusiasm and confidence. An appliance like this could really boost community involvement.

I think once you have completed this, perhaps the next step could be to consider a 'desktop' appliance version. I think the easier and closer to what people are familair with the lower the bar is for involvement. And reality is that regardless of what you and I do, most are using Windows.

[edited to tidy up some grammar etc]

Here are some extra ideas I'm working on for the next release of this patch:

- Build the web patchtool app to apply patches in an easy way (50%). You'll be able to select any of the images available under /srv/tklpatch/base-images and any of the patchs under /srv/tklpatch/patches and hit the "apply" button. It will take you to a page showing the log of the operation. This is my goal for this first version. For future versions, official feedback is going to be necessary, so I'll wait for the dynamic duo to stop fight crime and give their valuable feedback. 

Future ideas:

- Define a descriptor file for patches. This idea came to me while working on the patchtool. If the patch contains some meta data in a descriptor file (like the base image), it could be as simple as downloading the patch and upload it to the patchtool. Then the patch tool will read the meta data and automatically set the base image to use to apply the patch. If the image is not available the patchtool will ask you if you want to download the image (running the tklpatch-getimage command). The tool could even receive the url of the patch and download it, analize the data, and apply if everything is in place! This I hope would make it VERY EASY for newbies.  

When I download and rename this patch with WinXP I am having the same issue as I've had with other patches such as the issues I had with Adrian's Bacula patch. Something is broken somewhere and causing corruption! Here is the response from TKL Core when trying to apply this patch:

root@core ~# tklpatch-apply / tkldevenv.tar.gz
TKLPATCH_ISOLABEL: tkldevenv
# extracting patch tkldevenv.tar.gz to /tmp/tmp.nlrnBX4icb
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Exiting with failure status due to previous error

If I uncompress it in Windows, upload the folder, fix executable permissions and rebundle it with TKLPatch, it's fine!

The feedback I'd like more is on the project itself, but I can understand that with all that audio problem the attention got deviated to the screencast itself! I'm happy you're finding it helpful. I'll contribute with the documentation and tutorials, if this appliance make it's way to the official library. 

I made this one with RecordMyDesktop. As you say on the emails, the state of video editing on linux still sucks. Seems that every program has it's bugs. I wanted to use Xvidcap but didn't know how to record the audio in a separate track. So for easiness, I decided to go with RMD. But this is really an art, I had several shots to get to this. After that, I tried without success to remove the noise with audacity. So that's way I prefered to leave it that way, as this is a very "internal" screencast, not meant for production.

So for the second part, I'll do it recording separate sound & video. 

Bigger Fonts: I did use bigger fonts when I started, but my first attempt ended without positive results, besides learning. So when I did it the second time (about 2 days later), I forgot to set the bigger fonts. Duh!

I'll surely make some better screencast to add to the appliance. 

The mic is good, I used a Microsoft (please don't look me like that, M$ does good hardware and with total linux compatibility!) usb headset which I think is the right tool for the job. My lesson learned here was to minimize ambient noise at the mixer before starting to record. Also I think the cell phone had part of guilty, it seems that when he starts to fetch emails or sms it creates a lot of static interference. 

Another way to leverage the video would be to embed a very lightweight web server in the TKLPatch appliance and put up a static HTML page in the index that embeds the video and links to other resources...

BTW, how did you create the screencast? The loud static does makes it difficult to follow what you are saying at the start. Around 1:14 it does clear up though somewhat. Don't worry about not getting it right the first time. It took me a few hours to perfect the techniques I used for my first screencast about a year back.

Some random ideas for improvement:

1) Make sure the microphone is connected properly. If it is, you might want to invest in a new one that can record cleanly without so much static. I bought one for like $5 locally that was fine for voice.

2) Record the video and sound separately. I use xvidcap for video and Audacity for the audio. You can clean up the static and background noise by applying a filter.

3) Use bigger fonts in your terminal and record a smaller area of the screen. That way the screencast could be readable even in 360p, and HD mode would use less bandwidth.

4) Script it in advance and then edit the video for brevity. By editing out the parts where nothing much happens I usually cut the video time by 2-3X.

5) Start from the end: grab the viewer's attention by showing the "punch line" right at the start (e.g., like a title), then you can go into details and elaborate how stuff works.

Anyhow I wrote a bunch of notes for a blog post I haven't gotten around to publishing yet, I'll send you the draft by e-mail. Hopefully you'll find it useful!

Hi guys, I've made an screencast commenting on this appliance's console commands. You can watch it here if you're interested in the progress of this appliance. Things to consider:

- It's my first screencast ever. So I had some audio issues. Excuse me about the noise in the beginning of the video and later. Be patient, it's not so bad on the most of the video. I hope to get better on this. 

- Sorry for my english. I must admit I was nervious, and english is not my native language. So if I pronounce things in a funny way, I apologize!

- Watch it in HD fullscreen for readability of the command line. 

This screencast precedes the preview of the web interface. It's important to look at it first. I'll prepare the next one for this weekend. 

I hope this helps to get more feedback on this project. Please comment here in the forums please, I didn't found a way to disable comments on youtube. 

TKLDevEnv Technology Preview Pt.1

I did have a look at it but found the noise very distracting. Besides the noise I also had the problem that when I turned it up so I could hear it clearly my little subwoofer was distorting so I went off on a tangent trying to work out how to adjust the bass in Ubuntu (BTW I never did work that out - let me know if you know...)

My first confession is that I'm not really a fan of screencasts myself. I much prefer a clearly laid out and logical written tutorial. Obviously if its using a GUI then screenshots to help clarity are very useful but since my conversion to Linux I have decided that its much easier to do things at the CLI. The irony is that while newbs are often scared off by CLI, IMO its actually much more idiot proof than using a GUI because its just a case of copy-paste rather than "click this button, navigate to that tab, etc".

Having said that I realise that just because something works for me, doesn't mean that it'll work for everyone. The elitist atitude of Linux users in the past has been one of the biggest hurdles for newbs to overcome. So I think a screencast is a good idea, just that I'm probably not a good person to be reviewing it! :)

My son who is a real product of this 'digital age' is a big fan of screencasts. He uses them all the time to figure stuff out and learn about different techniques and technologies etc. SO perhaps I will get him to have a look at it.

I was about to add the comment that the submenu is one of the last aditions to the app (it's been really agile this project). So yes, I'm going to add the tab visual aid, it just that I haven't done it so far. The colors will surely change too. And the font's size in some places. It still needs a lot of work. 

Thanks for commenting! I missed your comments on the noisy screencast!

And even though the GUI isn't really pretty I've seen worse as an end product!Perhaps you already had this planned, but my only suggestion would be to define which section/subsection you are in, perhaps by shading the words of the links at the top, or making it look like a tab? That way you could see at a glance where you are.

Awesome!

IMHO, the main selling point of a wiki is that it's easy to edit collaboratively, whereas a forum topic has an owner. But I don't think we need to worry about that yet.

The particular wiki we are using has another advantage in that it's easier to browse thanks to the built-in navigation and support for hierarchical organization of pages. If I need to find a particular forum topic I admit I'll usually just search for it. Trouble is if I was a newbie I might not even know what to search for.

Yep, the TKLPatch editor is in the design, but not for the first release, so I haven't reach it. The initial design had only the patchtool. My goal is to provide the patchtool working for the first release. This is a great value I think, because users with no linux experience (which I calculate like 70% of the current userbase for what I read in the forums) will have a way to apply TKLPatches without having to confront with the "black screen". It'll be easier for them to patch and try new developments. The current procedure is very complex for the average windows user. I see it like a TKL Factory, where you can produce new isos. 

So once the patchtool works completely (tip: it does work already, but I want to add the actions to download the patched isos, delete them, upload/download from url tklpatches, and the tklpatch.info file) then I'll tune the interface, and the first release could go out with the simple (but functional) interface. 

Next I'll focus on the TKLPatch editor, which is almost a project by itself. But I can advance you it would be tab based, you'll hit the "edit" action in the projects page, and get to an interface with the following tabs: Info, Conf, Overlay, Packages. I haven't thought a lot about it, but the editor part will have bash color syntaxing in conf (the editor could be FCKEditor), maybe a widget for code snipplets (ie create mysql database, with a mini form with user/pass, and when you hit add it produces the code to create the database and user), the Info is for creating the tklpatch.info (a standard file with metadata I think would help to define the base image and author inside the patch), Overlay will let you upload files to the overlay system (although it's easier to simply access the share), and Packages will let you load debs. 

We'll see how it goes, as while programming things may take a different path. I'll try to write down a roadmap of the upcoming features. Do you think I should move the info about this project to the dev wiki? 

Also, I've make magic to be able to get the list of iso images, but when Lucid is finally out, I hope we can work on a safer way to get the image list and urls. (I'm proud of my tklpatch-getimage --update code but it could break if things change in the future)

I was curious as to the UI direction for the TKLPatch editor, but from the screenshots it doesn't look like you've reached that yet (if that is even a part of the initial design at all).

Don't worry about the looks at this point. Get the core functionality working well first. That's a measurable, easily defined goal. Improvements to the styling and design are open ended so it's possible to tweak things to no end, but more importantly you can work on that separately and going back to my blog post from last week, it's better to minimize the amount of context switches you have to do in your head.

You seem to be going in that direction anyway but I'll throw my 2cents in there anyway. Focus on the minimum viable product first - a core set of functionality that is useful, and improve it incrementally rather than trying to add a whole bunch of bells and whistles out the gate. In my experience anything that works well evolved from a smaller, simpler thing that worked well.

Organically improving something that already works is also much better for morale.

As I haven't had the time to do the screencast, here are some screenshots of the progress of the web interface. Please understand I haven't worked yet on a better design. The gui doesn't use javascript. In the final version, jquery will (I hope) do some magic to show you a modern gui as expected. My focus right now is that the application work. 

http://www.flickr.com/photos/55972691@N05/sets/72157625280045279/detail/

First let me tell you that I really appreciate you taking time to look at the project advancement. I'm very excited about recieving some feedback after so much time.

tklpatch checking conf script: I completely agree with you, I've falled on that path too, ending with an unmodified ISO. I'll add some sanity checks before processing the patch. 

The whole /srv directory should have nouser:nogroup ownership. This because of the samba share, otherwise you won't be able to delete/modify the files directly in the share. Now, regarding apache, I think I should add to the patch permissions for those folders to 777 but I'm not sure is this is the better way to do it. I'll check how I have mine. 

To change apache user, you can simply edit /etc/apache2/envvars and change the user and group from www-data to root. 

BTW I haven't test this approach! because I went to trying to modify tklpatch to run without root. So please confirm to me that this change will let you apply patches via the webapp. 

Thanks Adrian, had a few issues along the way. Firstly forgot to unpack the patch first (downloaded tarball from git rather than using git from TKL, which in retrospect was probably a bit silly...) Then the good 'ol non-executable conf file! Finally got it to patch all good (first few times I just ended up with a renamed Core ISO!)

So first bit of feedback (although not for you Adrian) I really think TKLPatch needs to check that the conf file is executable and if not error rather than merrily carry on repacking the virtually unchanged ISO.

But now it's working so that's great. I haven't had much time to play but I'll definately be back to this very soon - I have something in mind which I'll share when the time is right. ;)

First observation is that it won't let me upload ISOs or patches (permissions) but as you said it's only going to work properly with apache running as root anyway. BTW how do you do that? I had a bit of a google and other than interesting info about apache, the only info I found about running apache as root suggested compiling from source - surely there's an easier way! In the end I changed the file permissions on /srv. That fixed the upload issue but not sure about patching?

Maybe you already discovered this (sorry for a late reply) but the main patch downloads and install the web interface. Remember that: the webapp fails patching because of rights problems. You could fix this easily in this version configuring apache to run as root. As this is pottentially dangerous to security, I was trying to modify tklpatch to use fakeroot and fakechroot. I wasn't successfull in my tests, so I ended up planning to implement the use of lxc as suggested by Alon in an email. 

I'm also not sure about the security problems anyway, as this is a development environment, so I would really appreciate if you try the app and give some feedback. My main workflow is like this:

1. Start TKLDEVENV

2. Create a project from console with tklpatch-create (not implemented yet in the webapp)

3. Connect to the projects share. 

4. Start editing/copying files to overlay. 

5. Once ready I close editors to prevent temp files get into the patch.

6. From console, I build the patch (tklpatch-build)

7. From the webapp (running apache as root) you should be able to patch the base. 

8. If your base appliance is not in the tkldevenv, you can copy it to the baseimages share. If it's not downloaded, you can donwload it from the webapp. 

9. You can grab the output from the share. Or download it from the webapp. 

Remeber always to build the patch when making changes. The webapp is of course pretty inmature, but I didn't want to invest more time in it without validating it a bit with other users. Also, my main goal for the first release was to be able to apply existing patches. So features like uploading a patch to the appliance, and downloading baseimages took priority before editing/building the patches. 

I think I'll retake this project again playing with the lxc version.  

Thanks for taking time to take a look! Your feedback will be invaluable to me. 

Hi Adrian, I'm just building a TKLDevEnv from your GitHub repo. Would I be correct in assuming that the WebApp is already included or do I need to add it somewhere/somehow? If so where/how?

I've just been editing the same post and hadn't realised that you replied. Thanks for the quick reply, you rock!

I don't understand what was meant by running commands under XML-RPC either...

And I've heard Liraz complain about Perl before too so I certainly don't grudge you that. And you are right, Webmin is all about Admin, TKLDevEnv is all about newb friendly dev work, so it makes more sense for it to have it's own thing.

I had a bit of a look at Lighttpd but it too has checks so as not to run under root :( (I can see a pattern emerging here, I suspect that all web servers in the Ubuntu repos will have this security check) I think I might try compiling it from source and see how hard that is. Although it's probably not the ideal solution, it's the one I'm most likely to succeed with!

I successfully managed to compile LigHTTPd so it will run as root, but then I struggled to get it working with your WebUI :(

So I had a go at compiling Apache. Unfortunately there is something wrong with the Apache source package in Ubuntu (or something in my setup) as it won't compile, even without making any changes :( So all in all I've hit a wall on that front. I spent heaps of time yesterday and can't for the life of me work out what the problem is. :(

So next I will have a go at your suggestions re creating a simulated instance. Also I will do more testing from the commandline.

Out of interest, does this current version also include the OVZ conversion thingo? I know I could just test it and see but I'm busy doing other stuff at the moment and am being lazy!

XML-RPC is a webservice, but I'm not sure what they meant. Anyway, we can't definitely run apache as root. That was a test I was pending to do. What you can do right now to have a full experience of what was done:

service apache2 stop

Edit /opt/tkldevenv_webapp/urls.py and uncomment the following lines:

(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
       {'document_root': settings.STATIC_DOC_ROOT}),

./manage.py runserver your-ip-address:80

Ok changed the file permissions to nobody:nogroup (assume that's what you meant as there is no 'nouser' user, only nobody). But apache won't run as root, I just get this error:

Syntax error on line 145 of /etc/apache2/apache2.conf:
Error:\tApache has not been designed to serve pages while\n\trunning as root.
There are known race conditions that\n\twill allow any local user to read any file
on the system.\n\tIf you still desire to serve pages as root then\n\t
add -DBIG_SECURITY_HOLE to the CFLAGS env variable\n\tand then rebuild the server.
\n\tIt is strongly suggested that you instead modify the User\n\tdirective in
your httpd.conf file to list a non-root\n\tuser.\n

After changing it back it runs fine - line 145 is: User ${APACHE_RUN_USER}

Seems that we will need to find another way.

A little research turned up a few ideas:

1. Build apache from source.
2. To increase security, could leave apache as-is but run a 2nd webserver as root listening only on localhost (and pass commands from one server to the other).
3. Use XML-RPC (not 100% sure what this is? May be related to 2 above? Quite a few people talk about using it to run scripts as root).
4. Use some sort of intermediatry helper program (maybe related to 3 above?) Some have spoken about using a helper script that can be executed by www-data but runs as root (no idea how you'd do this).
5. Write jobs to a queue and have a cron job check for this queue and execute as root.

None of them are particularly appetising to me. I also note that with the file permissions set as-is (nobody:nogroup) in /srv the upload function does not work when apache is running under www-data.

[edit/update] Ok I've had another thought... Instead of reinventing the wheel, how about leveraging Webmin? I had a quick look and it's way above my pay grade, but what about a TKLDevEnv Webmin module? Obviously Webmin runs as root!?

Even if you didn't want to go that way, IIRC in the legacy appliances Webmin ran under lighttpd (but from what I can gather it now has it's own 'miniserver') so why couldn't we do that? This option may be something that I could do, I'll have a little look.

[more edit/update]I've been having a bit of a look at Webmin and it's pretty clever. It seems that even if you install it under a normal user account as along as the user is a sudo user it can use this to gain full root access tothe server! So... not sure how they do it, but about somehow copying how Webmin does it?