TurnKey Linux Virtual Appliance Library

TKLPatch for TKLPatch Development Environment v0.1 [Updated 07/11/2010]

Hi everyone. The following patch is an idea I had while developing patches for the dev contest. It aims to build a TKLPatch development environment, making easy the life of developers to produce new patches for the project. Some discussion have been taken here. This patch is in early stage, as there are a lot of ideas for this one, but community feedback will be important to get things done right. 

[Update 07/11/2010]

 

v0.1
- Added verification of existing iso to tklpatch command (to prevent error at the last step of the process)
- Added verification of existing iso in /srv/tklpatch/base-images to tklpatch-getimage command
- New --update option to tklpatch-getimage to download the lastest list of iso images directly from sourceforge. 
- Changed download directory for tklpatch-getimage to /tmp
- Various bash autocompletion fixes
- Created command tklpatch-edit for quick editing patches on the command line. 
- Starting to track versions.

 

[Update 08/10/2010]

Added tklpatch-create and tklpatch-build commands
Fixed a but in tklpatch bash autocompletion
Added two new shares: patches and projects
tklpatch now generates a log in /var/log/tklpatch/tklpatch-patchname.log
[/Update]
 
Right now, these are the first features of this appliance:
 

Features

Official tklpatch package pre-installed 
Modified versions of tklpatch and tklpatch-chroot scripts (changes explained below)
New commands: tklpatch-getimage tklpatch-publish, tklpatch-create, tklpatch-build, tklpatch-edit (explained bellow)
Samba shares for projects, patches, finished patched-isos and base-images (convenience)
Squid proxy optimized for package and archives downloading. (speed)
Bash autocompletion for tklpatch, tklpatch-getimage and tklpatch-build commands. 
 

Explanation:

Dont know how to set a patching environment to test the last patch published in the forums (that you need)? Simply deploy this appliance (ok, right know you have to apply it as a patch but in the future this could become official) and you'll have a complete patching environment ready to deploy and patch. 
 
But how do I get the base image to the patching environment? what's a scp? don't worry, if you already downloaded the base image, simply access the shared folder at //IP/baseimages and copy your base image there. If you haven't donwnloaded yet, you can use the tklpatch-getimage command to get the one you need. 
 
tklpatch-getimge --help # to get usage instructions
tklpatch-getimage --list # to get a list of available images
tklpatch-getimage turnkey-core-beta-lucid-x86 # get the turnkey-core lucid beta image. 
 
You can use TAB to autocomplete the name of the image you need to download. 
At this time of development, the imagelist is in the file /usr/local/share/tklpatch/baseimagelist (made of donwload urls). In the future this can be fetched from the TKL site itself. I've only included 4 images as this is a proof of concept. 
 
Messy development environment? lot of images in the current folder? Which one is that -patched image? No more messy environments! your base images (downloaded with tklpatch-getimage or copied) are located in /srv/tklpatch/base-images. Bash autocompletion makes it easy to get the image you need to patch simply using the TAB after tklpatch command. tklpatch itself have been modified to look for images in the current directory, but if no one is found, it automatically search for them on /srv/tklpatch/base-images. Also, now tklpatch adds the patch name to the final image, so if your running the asterisk patch, you'll final image will have -asterisk in it's name. 
 
Tired of waiting for package downloading each time you run your patch to test it? You are developing 3 patches that use apache and mysql and all of them take time downloading the same stuff? Wait no more! This dev env has an integrated squid optimized for package downloading. You don't have to invent your own ways to save time, as tklpatch-chroot have been modified to add the internal proxy to apt-get and wget, so all operations regarding package installations from ANY source are cached, as all files you get from wget. After patching, no trace of the proxy use will be left in your final iso. 
 
I've finished patching, now how do I get the final image to my computer? Easy: run the tklpatch-publish command to move the iso image to the predefined shared directory. Then access the patchedisos share to get it. 
 
You can access the shares from windows as any shared folder in the windows world. In linux, if you use ubuntu, the fastest way to access is symply type in nautilus:
smb://IP/baseimages/
smb://IP/patchedisos/
 
[Update 08/10/2010]
Want to star a new patch? simple. Use the tklpatch-create command to create a basic patch structure under /srv/tklpatch/projects. Simply type
 
	tklpatch-create mynewproject
 
The directory structure is going to be like this:
+ /srv/tklpatch/projects/mynewproject (directory)
+ /srv/tklpatch/projects/mynewproject/conf  (Conf file based on a customizable template located at /usr/local/share/tklpatch/conftemplate)
+ /srv/tklpatch/projects/mynewproject/debs (directory)
+ /srv/tklpatch/projects/mynewproject/overlay (directory)
+ /srv/tklpatch/projects/mynewproject/overlay/etc/confconsole/usage.txt (usage.txt file based on a customizable template located at /usr/local/share/tklpatch/usagetemplate.txt)
+ /srv/tklpatch/projects/mynewproject/overlay/etc/
 
Then, you can access that shared folder (\\IP\projects) to edit your conf script with your favorite editor, and add files easily to the debs and overlay directories. Once you have finish patching, you'll want to build your patch to end with a tar.gz to publish. Issue the command 
 
	tklpatch-build mynewproject 
  
And get your bundled patch automatically created at /srv/tklpatch/patches/mynewproject/datetime/mynewproject.tar.gz. You'll always keep the previous builds in case you need to go back to a previous version of the patch. For simplicity, you'll have a symlink to the last version located at /srv/tklpatch/patches/mynewproject/mynewproject.tar.gz (access this share through \\IP\patches). 
 
tklpatch bash autocompletion have been configured to search for patches under /srv/tklpatch/patches so you can use TAB to autocomplete the patch name! Also, if you want to check how the patch process went, you can access a log under /var/log/tklpatch/tklpatch-mynewproject with the complete output from the last time you applied the patch. 
[/Update]
 
[Update 07-11-2010]
 
I've worked on the tklpatch-getimage command, so that when you use the --update option, the command will fetch the file list from the sourceforge download page. The command automatically cleanup all the garbage and update the baseimagelist file to be useful for this appliance. So you can know access the whole list of files! this was done as I needed this command to fetch the files for the upcoming web patchtool. Also made several bug fixes. 
 
I also added tklpatch-edit command for fast editing on the command line (advanced users). It's going to use the system's default editor as per the $EDITOR env variable. 
 
I started to make a screencast, but it wasn't as easy as I thought! I get stuck looking for words when I'm talking, so I get nervious hahaha but I promise to have one the following week, as I wan't you guys giving feedback on this and I know it's time consuming to patch/install again with every new release. I wish to show you a session working with this appliance. 
 
I'm also about to release the first alpha version of the web patchtool. It looks very cool (to me). It's still very basic, and the ui design is very rough. I'm not using javascript as I plan to use some unobstrusive jquery for the second release to enhance user experience. 
 
I'm also starting to include version in the tar.gz for my own control.
[/update]
 
If there is interest in this, I'd like to give it a nice and catchy name. I hope to get ideas from the community. These are some of mine:
 
TKL-DEV: Not very catchy.
TKL-DEA: TurnkeyLinux Development Environment Appliance? 
TKL-ADE: TurnkeyLinux Appliance Development Environment?
 

What this patch does:

 
1. Set Hostname
HOSTNAME=tkldevenv
echo "$HOSTNAME" > /etc/hostname
sed -i "s|127.0.1.1 \(.*\)|127.0.1.1 $HOSTNAME|" /etc/hosts
hostname tkldevenv
 
2. Update package information
apt-get update
 
3. Install required packages. Note that I'm not installing webmin modules here because this system is for development/patching so ideally you won't be managing the squid or samba via webmin.
 
install tklpatch squid-deb-proxy samba
 
4. Backup original tklcommands. They are being replaced with the overlayed new ones in /usr/local/bin
mv /usr/bin/tklpatch /usr/bin/tklpatch.bak
chmod -x /usr/bin/tklpatch.bak
mv /usr/bin/tklpatch-chroot /usr/bin/tklpatch-chroot.bak
chmod -x /usr/bin/tklpatch-chroot.bak
5. Set squid port to a non default port and comment acl. If you ever need to make a patch using the squid-deb-proxy package, it won't have any problem starting as I'm setting the port to 12322 in this one. Also, removed a restriction to only cach debian repositories, as we want to use this one to also cache files donwloaded by our patch. 
 
sed -i "s/http_port 8000/http_port 127.0.0.1:12322/" /etc/squid-deb-proxy/squid-deb-proxy.conf
sed -i "s|acl to_ubuntu_mirrors dstdomain \"/etc/squid-deb-proxy/mirror-dstdomain.acl\"|#acl to_ubuntu_mirrors dstdomain \"/etc/squid-deb-proxy/mirror-dstdomain.acl\"|" /etc/squid-deb-proxy/squid-deb-proxy.conf
sed -i "s/http_access deny \!to_ubuntu_mirrors/#http_access deny \!to_ubuntu_mirrors/" /etc/squid-deb-proxy/squid-deb-proxy.conf
 
6. [Updated 08/10/2010] Create directories for the samba shares. Configuration of the shares is being overlayed.
mkdir -p /srv/tklpatch/patched-isos
chown nobody.nogroup /srv/tklpatch/patched-isos
mkdir -p /srv/tklpatch/base-images
chown nobody.nogroup /srv/tklpatch/base-images
mkdir -p /srv/tklpatch/patches
chown nobody.nogroup /srv/tklpatch/patches
mkdir -p /srv/tklpatch/projects
chown nobody.nogroup /srv/tklpatch/projects

7. Stop services 

service squid stop
service squid-deb-proxy stop
service squid-deb-proxy-avahi stop

Diff for tklpatch [Updated 08/10/2010]

2a3
> # Modified by Adrian Moya  2010
67c68
< if ! [ -f $isofile ]; then
---
> if ! [[ -f $isofile || -f /srv/tklpatch/base-images/$isofile ]]; then
71c72
< if ! [ -e $patch ]; then
---
> if ! [[ -e $patch || -e /srv/tklpatch/patches/$patch/$patch.tar.gz ]]; then
79a81,83
> patchname="$(basename $patch)"
> patchname="$(echo $patchname | sed 's/.tar.gz$//')"
> 
83,86c87,100
< tklpatch-extract-iso $isofile
< tklpatch-apply $rootfs $patch
< tklpatch-prepare-cdroot $rootfs $cdroot
< tklpatch-geniso $cdroot
---
> # check if files are local or in /srv/tklpatch
> if ! [ -f $isofile ]; then
>     isofile=/srv/tklpatch/base-images/$isofile
> fi
> if ! [ -e $patch ]; then
>     patch=/srv/tklpatch/patches/$patch/$patch.tar.gz
> fi
> 
> DATETIME=`date +%d-%m-%y\ %H:%M:%S`
> echo "tklpatch run at $DATETIME" > /var/log/tklpatch/tklpatch-$patchname.log
> tklpatch-extract-iso $isofile | tee -a /var/log/tklpatch/tklpatch-$patchname.log
> tklpatch-apply $rootfs $patch | tee -a /var/log/tklpatch/tklpatch-$patchname.log
> tklpatch-prepare-cdroot $rootfs $cdroot | tee -a /var/log/tklpatch/tklpatch-$patchname.log
> tklpatch-geniso $cdroot $name-$patchname.iso | tee -a /var/log/tklpatch/tklpatch-$patchname.log
91a106
> echo "Log created at /var/log/tklpatch/tklpatch-$patchname.log"

Diff for tklpatch-chroot

2a3
> # Modified by Adrian Moya <adrianmoya@gmail.com> 2010
195a197,208
> 
>     if ! [ -f $root/etc/apt/apt.conf.d/02proxy ]; then
>        echo "Acquire::http { Proxy \"http://localhost:12322\"; };" > $root/etc/apt/apt.conf.d/02proxy
>        traphook="${traphook}rm $root/etc/apt/apt.conf.d/02proxy;"
>     fi
>     
>     if ! [ -f $root/etc/wgetrc.bak ]; then
>         cp $root/etc/wgetrc $root/etc/wgetrc.bak
>        echo "use_proxy = on" >> $root/etc/wgetrc
>         echo "http_proxy = http://localhost:12322/" >> $root/etc/wgetrc
>        traphook="${traphook}mv $root/etc/wgetrc.bak $root/etc/wgetrc;"
>     fi   

Conf script default template:

#!/bin/bash -ex
# By Author  

install()
{
    DEBIAN_FRONTEND=noninteractive apt-get -y \
        -o DPkg::Options::=--force-confdef \
        -o DPkg::Options::=--force-confold \
        install $@
}

# Set Hostname
HOSTNAME=
echo "$HOSTNAME" > /etc/hostname
sed -i "s|127.0.1.1 \(.*\)|127.0.1.1 $HOSTNAME|" /etc/hosts
hostname 

#Update package information
apt-get update

# Install required packages
install

# Clean apt cache
apt-get clean

Future ideas:

I have lots of ideas for this one, and hope to polish them with community feedback. 
- Directory dedicated to patches themselves? /srv/tklpatch/patches? [Done 08/10/2010]
- Create a new project, versioning of patches [Done 08/10/2010]
- Web application to patch. (using the new commands as a base). 
- Web application to patch development? Edit conf file via web, add overlayed files via the web app, etc.
 
Comments welcome!
Attached: 
Jeremy's picture

Nice work Adrian

This is a great idea. The only thing I thought as reading you beat me to! (Under 'future ideas') a share for the patches themselves (to load and or to get out once developed). Or maybe they could go in the other folders with the isos?

Regardless, good work and I'll have to have a play with this sometime real soon.

Basil Kurian's picture

nice appliance

The thing which I like most in this patch is caching provided by squid-proxy ..... will it work for svn,git and mercurial repos ? Also is there any way by which caching can be made in gem install commands ?

Regarding the name , I prefer  "TKL-ADE: TurnkeyLinux Appliance Development Environment"

How about adding zsync to update Core image to latest one by downloading the changed bits only [http://www.webupd8.org/2009/10/how-to-upgrade-your-karmic-iso-with.html] . I know it requires a zsync server on turnkeylinux website.


why not use FQDN ?

I found change the hostname, but why not use FQDN? some program like mail would need use .


I hope that the new TKL Core let you

change the hostname on installation. Let's hope so, as it is necesary to be able to customize the FQDN for many appliances. Also regional settings would be nice. 

Some extra ideas

Here are some extra ideas I'm working on for the next release of this patch:

- Build the web patchtool app to apply patches in an easy way (50%). You'll be able to select any of the images available under /srv/tklpatch/base-images and any of the patchs under /srv/tklpatch/patches and hit the "apply" button. It will take you to a page showing the log of the operation. This is my goal for this first version. For future versions, official feedback is going to be necessary, so I'll wait for the dynamic duo to stop fight crime and give their valuable feedback. 

Future ideas:

- Define a descriptor file for patches. This idea came to me while working on the patchtool. If the patch contains some meta data in a descriptor file (like the base image), it could be as simple as downloading the patch and upload it to the patchtool. Then the patch tool will read the meta data and automatically set the base image to use to apply the patch. If the image is not available the patchtool will ask you if you want to download the image (running the tklpatch-getimage command). The tool could even receive the url of the patch and download it, analize the data, and apply if everything is in place! This I hope would make it VERY EASY for newbies.  

Jeremy's picture

Wow! This sounds like an awesome appliance!

This sounds so cool Adrian. I'm excited about the possibilities! You've obviously put a lot of thought into this! These ideas will make community involvement so much easier.

I think that is a really good thing. Having this available as an appliance will  potentially allow for some niche community interests to be accommodated without requiring dev's approval and/or support. Another advantage of making patching more accessable, is that hopefully it will get more people involved in patching and development. Often all it takes is some small successes to buoy people's enthusiasm and confidence. An appliance like this could really boost community involvement.

I think once you have completed this, perhaps the next step could be to consider a 'desktop' appliance version. I think the easier and closer to what people are familair with the lower the bar is for involvement. And reality is that regardless of what you and I do, most are using Windows.

[edited to tidy up some grammar etc]

Jeremy's picture

I'm getting the same issue with this patch as I did with others

When I download and rename this patch with WinXP I am having the same issue as I've had with other patches such as the issues I had with Adrian's Bacula patch. Something is broken somewhere and causing corruption! Here is the response from TKL Core when trying to apply this patch:

root@core ~# tklpatch-apply / tkldevenv.tar.gz
TKLPATCH_ISOLABEL: tkldevenv
# extracting patch tkldevenv.tar.gz to /tmp/tmp.nlrnBX4icb
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Exiting with failure status due to previous error

If I uncompress it in Windows, upload the folder, fix executable permissions and rebundle it with TKLPatch, it's fine!

Screencast for the command line commands

Hi guys, I've made an screencast commenting on this appliance's console commands. You can watch it here if you're interested in the progress of this appliance. Things to consider:

- It's my first screencast ever. So I had some audio issues. Excuse me about the noise in the beginning of the video and later. Be patient, it's not so bad on the most of the video. I hope to get better on this. 

- Sorry for my english. I must admit I was nervious, and english is not my native language. So if I pronounce things in a funny way, I apologize!

- Watch it in HD fullscreen for readability of the command line. 

This screencast precedes the preview of the web interface. It's important to look at it first. I'll prepare the next one for this weekend. 

I hope this helps to get more feedback on this project. Please comment here in the forums please, I didn't found a way to disable comments on youtube. 

TKLDevEnv Technology Preview Pt.1

Liraz Siri's picture

Great idea, the only showstopper is the sound problem.

Making a screencast tutorial is a great idea! We've been planning on creating a section on the website for videos, we just don't have enough material yet to make that worthwhile, but this would be a nice addition.

Another way to leverage the video would be to embed a very lightweight web server in the TKLPatch appliance and put up a static HTML page in the index that embeds the video and links to other resources...

BTW, how did you create the screencast? The loud static does makes it difficult to follow what you are saying at the start. Around 1:14 it does clear up though somewhat. Don't worry about not getting it right the first time. It took me a few hours to perfect the techniques I used for my first screencast about a year back.

Some random ideas for improvement:

1) Make sure the microphone is connected properly. If it is, you might want to invest in a new one that can record cleanly without so much static. I bought one for like $5 locally that was fine for voice.

2) Record the video and sound separately. I use xvidcap for video and Audacity for the audio. You can clean up the static and background noise by applying a filter.

3) Use bigger fonts in your terminal and record a smaller area of the screen. That way the screencast could be readable even in 360p, and HD mode would use less bandwidth.

4) Script it in advance and then edit the video for brevity. By editing out the parts where nothing much happens I usually cut the video time by 2-3X.

5) Start from the end: grab the viewer's attention by showing the "punch line" right at the start (e.g., like a title), then you can go into details and elaborate how stuff works.

Anyhow I wrote a bunch of notes for a blog post I haven't gotten around to publishing yet, I'll send you the draft by e-mail. Hopefully you'll find it useful!

Liraz Siri's picture

Once we add TKLDevEnv to the library we can add this to the docs

Thought of another idea. TKLDevEnv looks much easier to get started with than raw tklpatch, especially when you throw in the video. So once we come out with the next release we could revise the community docs to include it, along with the tutorial video.

Ok, this is what I'm looking for

The feedback I'd like more is on the project itself, but I can understand that with all that audio problem the attention got deviated to the screencast itself! I'm happy you're finding it helpful. I'll contribute with the documentation and tutorials, if this appliance make it's way to the official library. 

Thanks for the comments and recommendations on screencasting!

I made this one with RecordMyDesktop. As you say on the emails, the state of video editing on linux still sucks. Seems that every program has it's bugs. I wanted to use Xvidcap but didn't know how to record the audio in a separate track. So for easiness, I decided to go with RMD. But this is really an art, I had several shots to get to this. After that, I tried without success to remove the noise with audacity. So that's way I prefered to leave it that way, as this is a very "internal" screencast, not meant for production.

So for the second part, I'll do it recording separate sound & video. 

Bigger Fonts: I did use bigger fonts when I started, but my first attempt ended without positive results, besides learning. So when I did it the second time (about 2 days later), I forgot to set the bigger fonts. Duh!

I'll surely make some better screencast to add to the appliance. 

The mic is good, I used a Microsoft (please don't look me like that, M$ does good hardware and with total linux compatibility!) usb headset which I think is the right tool for the job. My lesson learned here was to minimize ambient noise at the mixer before starting to record. Also I think the cell phone had part of guilty, it seems that when he starts to fetch emails or sms it creates a lot of static interference. 

Screenshots of progress

As I haven't had the time to do the screencast, here are some screenshots of the progress of the web interface. Please understand I haven't worked yet on a better design. The gui doesn't use javascript. In the final version, jquery will (I hope) do some magic to show you a modern gui as expected. My focus right now is that the application work. 

So this is pre-alpha v0.00001 of the app ;) Hope you like what you see and comments are welcome directly on flicker or in this thread

http://www.flickr.com/photos/55972691@N05/sets/72157625280045279/detail/

Jeremy's picture

Looking really good so far Adrian

And even though the GUI isn't really pretty I've seen worse as an end product!Perhaps you already had this planned, but my only suggestion would be to define which section/subsection you are in, perhaps by shading the words of the links at the top, or making it look like a tab? That way you could see at a glance where you are.

Awesome!

You beat me!

I was about to add the comment that the submenu is one of the last aditions to the app (it's been really agile this project). So yes, I'm going to add the tab visual aid, it just that I haven't done it so far. The colors will surely change too. And the font's size in some places. It still needs a lot of work. 

Thanks for commenting! I missed your comments on the noisy screencast!

Jeremy's picture

Sorry I didn't comment on your screencast

I did have a look at it but found the noise very distracting. Besides the noise I also had the problem that when I turned it up so I could hear it clearly my little subwoofer was distorting so I went off on a tangent trying to work out how to adjust the bass in Ubuntu (BTW I never did work that out - let me know if you know...)

My first confession is that I'm not really a fan of screencasts myself. I much prefer a clearly laid out and logical written tutorial. Obviously if its using a GUI then screenshots to help clarity are very useful but since my conversion to Linux I have decided that its much easier to do things at the CLI. The irony is that while newbs are often scared off by CLI, IMO its actually much more idiot proof than using a GUI because its just a case of copy-paste rather than "click this button, navigate to that tab, etc".

Having said that I realise that just because something works for me, doesn't mean that it'll work for everyone. The elitist atitude of Linux users in the past has been one of the biggest hurdles for newbs to overcome. So I think a screencast is a good idea, just that I'm probably not a good person to be reviewing it! :)

My son who is a real product of this 'digital age' is a big fan of screencasts. He uses them all the time to figure stuff out and learn about different techniques and technologies etc. SO perhaps I will get him to have a look at it.

Liraz Siri's picture

Awesome! Don't worry about how it looks yet

Adrian, very exciting to see this get closer to a working state! I'm a bit jealous as this looks like a fun project and I'm currently stuck on a bit of a boring assignment for work.

I was curious as to the UI direction for the TKLPatch editor, but from the screenshots it doesn't look like you've reached that yet (if that is even a part of the initial design at all).

Don't worry about the looks at this point. Get the core functionality working well first. That's a measurable, easily defined goal. Improvements to the styling and design are open ended so it's possible to tweak things to no end, but more importantly you can work on that separately and going back to my blog post from last week, it's better to minimize the amount of context switches you have to do in your head.

You seem to be going in that direction anyway but I'll throw my 2cents in there anyway. Focus on the minimum viable product first - a core set of functionality that is useful, and improve it incrementally rather than trying to add a whole bunch of bells and whistles out the gate. In my experience anything that works well evolved from a smaller, simpler thing that worked well.

Organically improving something that already works is also much better for morale.

Glad you like it!

Yep, the TKLPatch editor is in the design, but not for the first release, so I haven't reach it. The initial design had only the patchtool. My goal is to provide the patchtool working for the first release. This is a great value I think, because users with no linux experience (which I calculate like 70% of the current userbase for what I read in the forums) will have a way to apply TKLPatches without having to confront with the "black screen". It'll be easier for them to patch and try new developments. The current procedure is very complex for the average windows user. I see it like a TKL Factory, where you can produce new isos. 

So once the patchtool works completely (tip: it does work already, but I want to add the actions to download the patched isos, delete them, upload/download from url tklpatches, and the tklpatch.info file) then I'll tune the interface, and the first release could go out with the simple (but functional) interface. 

Next I'll focus on the TKLPatch editor, which is almost a project by itself. But I can advance you it would be tab based, you'll hit the "edit" action in the projects page, and get to an interface with the following tabs: Info, Conf, Overlay, Packages. I haven't thought a lot about it, but the editor part will have bash color syntaxing in conf (the editor could be FCKEditor), maybe a widget for code snipplets (ie create mysql database, with a mini form with user/pass, and when you hit add it produces the code to create the database and user), the Info is for creating the tklpatch.info (a standard file with metadata I think would help to define the base image and author inside the patch), Overlay will let you upload files to the overlay system (although it's easier to simply access the share), and Packages will let you load debs. 

We'll see how it goes, as while programming things may take a different path. I'll try to write down a roadmap of the upcoming features. Do you think I should move the info about this project to the dev wiki? 

Also, I've make magic to be able to get the list of iso images, but when Lucid is finally out, I hope we can work on a safer way to get the image list and urls. (I'm proud of my tklpatch-getimage --update code but it could break if things change in the future)

Liraz Siri's picture

Wiki best for collaboration with multiple authors

Don't worry about the right place to collaborate, just use what is most convenient. You'll know when you outgrow it because it will start to get really painful. When you are really feeling the pain, that's when you really need to think about alternatives. It seems the forums are working fine, and I don't see why they shouldn't continue to work fine so long as you are the primary developer.

IMHO, the main selling point of a wiki is that it's easy to edit collaboratively, whereas a forum topic has an owner. But I don't think we need to worry about that yet.

The particular wiki we are using has another advantage in that it's easier to browse thanks to the built-in navigation and support for hierarchical organization of pages. If I need to find a particular forum topic I admit I'll usually just search for it. Trouble is if I was a newbie I might not even know what to search for.

Jeremy's picture

What to do with tkldevenv_webapp?

Hi Adrian, I'm just building a TKLDevEnv from your GitHub repo. Would I be correct in assuming that the WebApp is already included or do I need to add it somewhere/somehow? If so where/how?

It's included already

Maybe you already discovered this (sorry for a late reply) but the main patch downloads and install the web interface. Remember that: the webapp fails patching because of rights problems. You could fix this easily in this version configuring apache to run as root. As this is pottentially dangerous to security, I was trying to modify tklpatch to use fakeroot and fakechroot. I wasn't successfull in my tests, so I ended up planning to implement the use of lxc as suggested by Alon in an email. 

I'm also not sure about the security problems anyway, as this is a development environment, so I would really appreciate if you try the app and give some feedback. My main workflow is like this:

1. Start TKLDEVENV

2. Create a project from console with tklpatch-create (not implemented yet in the webapp)

3. Connect to the projects share. 

4. Start editing/copying files to overlay. 

5. Once ready I close editors to prevent temp files get into the patch.

6. From console, I build the patch (tklpatch-build)

7. From the webapp (running apache as root) you should be able to patch the base. 

8. If your base appliance is not in the tkldevenv, you can copy it to the baseimages share. If it's not downloaded, you can donwload it from the webapp. 

9. You can grab the output from the share. Or download it from the webapp. 

Remeber always to build the patch when making changes. The webapp is of course pretty inmature, but I didn't want to invest more time in it without validating it a bit with other users. Also, my main goal for the first release was to be able to apply existing patches. So features like uploading a patch to the appliance, and downloading baseimages took priority before editing/building the patches. 

I think I'll retake this project again playing with the lxc version.  

Thanks for taking time to take a look! Your feedback will be invaluable to me. 

Jeremy's picture

Hehe...got there in the end!

Thanks Adrian, had a few issues along the way. Firstly forgot to unpack the patch first (downloaded tarball from git rather than using git from TKL, which in retrospect was probably a bit silly...) Then the good 'ol non-executable conf file! Finally got it to patch all good (first few times I just ended up with a renamed Core ISO!)

So first bit of feedback (although not for you Adrian) I really think TKLPatch needs to check that the conf file is executable and if not error rather than merrily carry on repacking the virtually unchanged ISO.

But now it's working so that's great. I haven't had much time to play but I'll definately be back to this very soon - I have something in mind which I'll share when the time is right. ;)

First observation is that it won't let me upload ISOs or patches (permissions) but as you said it's only going to work properly with apache running as root anyway. BTW how do you do that? I had a bit of a google and other than interesting info about apache, the only info I found about running apache as root suggested compiling from source - surely there's an easier way! In the end I changed the file permissions on /srv. That fixed the upload issue but not sure about patching?

Glad to know you installed the environment!

First let me tell you that I really appreciate you taking time to look at the project advancement. I'm very excited about recieving some feedback after so much time.

tklpatch checking conf script: I completely agree with you, I've falled on that path too, ending with an unmodified ISO. I'll add some sanity checks before processing the patch. 

The whole /srv directory should have nouser:nogroup ownership. This because of the samba share, otherwise you won't be able to delete/modify the files directly in the share. Now, regarding apache, I think I should add to the patch permissions for those folders to 777 but I'm not sure is this is the better way to do it. I'll check how I have mine. 

To change apache user, you can simply edit /etc/apache2/envvars and change the user and group from www-data to root. 

BTW I haven't test this approach! because I went to trying to modify tklpatch to run without root. So please confirm to me that this change will let you apply patches via the webapp. 

Jeremy's picture

More feedback

Ok changed the file permissions to nobody:nogroup (assume that's what you meant as there is no 'nouser' user, only nobody). But apache won't run as root, I just get this error:

Syntax error on line 145 of /etc/apache2/apache2.conf:
Error:\tApache has not been designed to serve pages while\n\trunning as root.  
There are known race conditions that\n\twill allow any local user to read any file 
on the system.\n\tIf you still desire to serve pages as root then\n\t
add -DBIG_SECURITY_HOLE to the CFLAGS env variable\n\tand then rebuild the server.
\n\tIt is strongly suggested that you instead modify the User\n\tdirective in 
your httpd.conf file to list a non-root\n\tuser.\n

After changing it back it runs fine - line 145 is: User ${APACHE_RUN_USER}

Seems that we will need to find another way.

A little research turned up a few ideas:

  1. Build apache from source.
  2. To increase security, could leave apache as-is but run a 2nd webserver as root listening only on localhost (and pass commands from one server to the other).
  3. Use XML-RPC (not 100% sure what this is? May be related to 2 above? Quite a few people talk about using it to run scripts as root).
  4. Use some sort of intermediatry helper program (maybe related to 3 above?) Some have spoken about using a helper script that can be executed by www-data but runs as root (no idea how you'd do this).
  5. Write jobs to a queue and have a cron job check for this queue and execute as root.

None of them are particularly appetising to me. I also note that with the file permissions set as-is (nobody:nogroup) in /srv the upload function does not work when apache is running under www-data.

[edit/update] Ok I've had another thought... Instead of reinventing the wheel, how about leveraging Webmin? I had a quick look and it's way above my pay grade, but what about a TKLDevEnv Webmin module? Obviously Webmin runs as root!?

Even if you didn't want to go that way, IIRC in the legacy appliances Webmin ran under lighttpd (but from what I can gather it now has it's own 'miniserver') so why couldn't we do that? This option may be something that I could do, I'll have a little look.

[more edit/update]I've been having a bit of a look at Webmin and it's pretty clever. It seems that even if you install it under a normal user account as along as the user is a sudo user it can use this to gain full root access tothe server! So... not sure how they do it, but about somehow copying how Webmin does it?

OK!

XML-RPC is a webservice, but I'm not sure what they meant. Anyway, we can't definitely run apache as root. That was a test I was pending to do. What you can do right now to have a full experience of what was done:

service apache2 stop

Edit /opt/tkldevenv_webapp/urls.py and uncomment the following lines:

(r'^site_media/(?P<path>.*)$', 'django.views.static.serve',
       {'document_root': settings.STATIC_DOC_ROOT}),
 
Execute the following (in /opt/tkldevenv_webapp)
./manage.py runserver your-ip-address:80
 
There you'll have the development server running on port 80 and you can simulate the devenv working as expected. 
 
Using queues was on my plans for later releases. They are going to be useful, and in a multiuser environment they will surelly be needed. So my plans was to rewrite the patching tool with queues. 
 
Finally, no webmin for me! its perl, and I have no clue about it (and no will to learn :P) Anyway, webmin is an administration frontend. Patching and editing patches are not administration tasks, so it doesn't make much sense to create a dev environment as a webmin module. But it's not a bad idea of using a light web server to run the app as root. 
 
For now, it needs some rewriting to find a way to solve this issue. Point 4 looks like the simplest solution. 
Jeremy's picture

Oops missed your reply

I've just been editing the same post and hadn't realised that you replied. Thanks for the quick reply, you rock!

I don't understand what was meant by running commands under XML-RPC either...

And I've heard Liraz complain about Perl before too so I certainly don't grudge you that. And you are right, Webmin is all about Admin, TKLDevEnv is all about newb friendly dev work, so it makes more sense for it to have it's own thing.

I had a bit of a look at Lighttpd but it too has checks so as not to run under root :( (I can see a pattern emerging here, I suspect that all web servers in the Ubuntu repos will have this security check) I think I might try compiling it from source and see how hard that is. Although it's probably not the ideal solution, it's the one I'm most likely to succeed with!

Jeremy's picture

No joy recompiling Apache

I successfully managed to compile LigHTTPd so it will run as root, but then I struggled to get it working with your WebUI :(

So I had a go at compiling Apache. Unfortunately there is something wrong with the Apache source package in Ubuntu (or something in my setup) as it won't compile, even without making any changes :( So all in all I've hit a wall on that front. I spent heaps of time yesterday and can't for the life of me work out what the problem is. :(

So next I will have a go at your suggestions re creating a simulated instance. Also I will do more testing from the commandline.

Out of interest, does this current version also include the OVZ conversion thingo? I know I could just test it and see but I'm busy doing other stuff at the moment and am being lazy!

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)