Beta of TurnKey Core on Debian Lenny (+ Ubuntu vs Debian)

Yes, you said explicitly what I was implying. Unit tests, black box testing, is best used for regression testing. It is also good for configuration and integration ("thread") testing.

For everything else you still usually need some kind of user testing, because you usually don't have the luxury of knowing in advance what is going to fail, otherwise you would have prevented that already.

I expressed my thoughts on #turnkey as follows:

<tinjaw> alonswartz, you say...
<tinjaw> "Frankly, we don't have the resources to thoroughly test both Debian and Ubuntu based builds of all TurnKey appliances."
<tinjaw> Fine. Spend the resources to build an automated test suite. Better yet...
<tinjaw> I am sure that the various applications that make up the appliances have their own test suites
<tinjaw> Turnkey would be wise to spend the time creating a meta test structure.
<tinjaw> Along with community based testing, that should produce a fairly solid product line.
<alonswartz> tinjaw: true
<alonswartz> up till now the appliances go through testing during development, and after building the ISO's, VM's, and EC2 builds
<alonswartz> but testing is more than making sure the software in the appliances work as expected
<alonswartz> we also need to test the integration of the parts
<tinjaw> that is where the meta testing would come into play
<alonswartz> I agree, if it could all be automated would be great - we could learn something from the ubuntu iso-testing team
<tinjaw> Now, I am in NO WAY committing, but ...
<tinjaw> I have felt that I *need* to learn (and use on my own projects) Selenium. So,
<tinjaw> Maybe working on a Selenium-driven test suite for TurnKey appliances ***might*** be something I would be willing to invest my time with.
<tinjaw> We might be able to even contribute some of that back upstream
<alonswartz> tinjaw: that would be great!
<alonswartz> if you could add a comment to the post on what we discussed, it might spark more discussion and input from the rest of the community

What does the rest of the community think about this?

But a big part of that advantage relies on the operating system level maintaining itself with security updates and such. Otherwise you do have to be aware of what components are in your appliance and make sure to update them yourself in case there are security issues.

That is, assuming you are using the appliance in circumstances where security matter and not just for internal testing and such.

Debian vs Ubuntu are closely related but they make different trade offs with regards to security.

Hi, I'm a Turnkeylinux fun, and I use the appliances a lot mostly as virtual appliances.

For me, it is the same if they use Debian or Ubuntu, since the most important part is the application that is mounted on the appliance, and if this works fine, it is not important for the end user like me what is behind... so I think it is a matter of preference of the developers (you!).

Thanks again for the great appliances.

Installed quickly to a VMWare virtual machine on the first install. It sluggish with one security update, but recovers immediately to bring up confconsole.

I used ssh to apt-get myself a lamp stack up and running. I then installed subversion and tried it out. Worked perfectly. What struck me was how responsive it was. Tomorrow I'll be working on bugs for an application in the virtual machine, so I'll be able to say more. So far, absolutely nothing for me to be concerned with.

Update: Web app installed with subversion and configured without issues. Es ist guile!

multiseat sounds very neat, if you can get it working. For some reason I never considered connecting multiple keyboard / mice via USB. It's exactly the kind of application that would be a good fit for TurnKey - easy to use once it's up but needing expert help to get there. I wonder if there's an easy way to detect extra sets of input devices and start X sessions that are associated to them only. If I'm not mistaken usually any input device works with everything. When the time is right, we can start working out the details in a new thread, hopefully with the help of someone who's done this before and has compatible hardware!

Recovery partitions - now I get it. Brilliant idea. Copying the ISO to the hard-drive and booting it from GRUB saves you the trouble of needing a CD/USB. But, it's less secure. An attacker that has access to the Windows end of the system will still be able to tamper with the ISO, or anything else on the hard disk for that matter (e.g., grub). For this to work you'll need to prevent the untrusted machine from accessing everything (e.g., running it under virtualization). A nice solution could be a  TKL meta appliance that's optimized for running desktop type operating systems (rather than just head-less) servers.

Multiseat - no extra hardware required beyond an additional kvm set (keyboard/video(monitor)/mouse) for each extra user (USB keyboard/mouse) and of course a video outlet per user on the PC. Most video cards are dual head these days, even entry level ones, so an average PC with a vid card will support 2 independant users without any additional outlay (other than kvm). Apparently its quite possible to have up to 10 concurrent users all using the same hardware. I think this could be a niche open source win with relevance for education, public access, kiosks and even family homes if an easy to setup, preconfigured distro was availble (read TKL) to install straight to hardware.

Live Tools - sounds like we're on the same page there then!

Meta Live Tool - That wasn't actually what I was thinking but that's probably an even better idea! AFAIK it is possible to have a CD boot GRUB2, then each Live tool could be a separate (.iso) entity. If GRUB will do that (which I've been lead to believe it will) then that approach could also be used to create TKL Meta "Demo" CDs with multiple versions of TKL. This could help tech savy/IT pros 'sell' TKL to potential clients and the general public. The beauty of having each install as an .iso is that they are each complete and independant, thus avoiding many of the potential problems that can occur with most All-In-One type solutions.

User Friendly Live Recovery 'partition' - was more what I had in mind. To be specific I was thinking a "Windows user friendly" Linux based recovery 'partition'. A value added feature that could be included with TKL Client OSs and/or possibly 'installed' (GRUB2 would need to be installed but the Live Recovery OS would simply reside as an .iso) alongside another OS. Stretching this idea even further, the Australian Federal Police last year made a (very carefully worded) statement urging all users of online banking to only do online banking using a Live environment (rather than an installed OS) to mitigate security concerns. Perhaps a TKL "Secure Online" Live Client that could run from CD, USB or be 'installed'?

Multiseat: is a concept I had never considered before. Very interesting but I imagine you would need special hardware to get it to work?

Live tools: Looking at the adhoc todo list in my notes I see that we have had some of the "live tools" ideas (e.g., "system rescue" type appliance) in there for a while now though I admit I hadn't taken the concept as far as you have. The first step in actually implementing this stuff would be to get a solid Core Client with a front-end first. It will then be easier for us (with the help of the community of course) to play with these ideas. Sort of "throw stuff out on the wall and see what sticks" type approach.

Meta live tool?: I admit I don't feel I've fully wrapped my mind yet around your ideas on a live appliance that can run other live appliances. Do you mean that in this case each appliance would be a sort of modular package in a system that can boot any of them? Then you roll the ISOs into a mega all-in-one live tool appliance?

I think you're right that Ubuntu may well be a better fit for these style of appliance. Although could there be room for Deian there to? Perhaps there may be some specialised client roles that are better fulfilled by a Debian base? Maybe security is particularly important for a specific client role? As long as they have inter-appliance continuity (by way of Core theme & apps) I think a blend of Ubuntu & Debian appliances may still work. It should also be relatively easy to achieve that continuity, as Debian and Ubuntu are so close. Perhaps with these client appliances, it be better to just select some to have a Debian twin? Rather replicating the whole range (I think the server appliances are somewhat different).

Off on a tangent; Multiseat is something I have been looking at over the last few months. I'd be great to have "TKL Multiseat Client"!

Perhaps stretching the vision further than you've been thinking; but I reckon a range of TKL 'live tool' appliances would fit nicely alongside the existing range. This too would be consistent with the TKL aim of bringing the best of open source to the public. TKL cleanup, backup,  recovery, HDD tools, offline antivirus/malware scanning (aimed at use with Win &/or Linux desktops) could all be great appliances.

Another potential plus for using Ubuntu as a base for TKL Clients is GRUB2. By leveraging GRUB2's ability to boot from .iso, a generic TKL 'live tool' appliance could be setup as an 'installed' (but not really) recovery/troubleshooting/live boot option for the "Client" appliances? This live environment could have packages updated leveraging TKLPatch, and upgraded to the latest release simply by downloading the new .iso (and updating GRUB2 menu - which ideally would all be handled by a script).

Sorry, waffling! :)

Client-side appliance-type applications would include any role-specific application that needs a front-end. Off the top of my head that would be stuff like media centers, kiosks, multi-media editing, thin clients, gaming emulators, and desktop virtualization platforms (e.g., VirtualBox).

For these sorts of applications, Ubuntu's usability enhancements might be more of an advantage.

Also, I may be showing my ignorance here but I've always found it a bit awe inspiring that the blind could overcome the inherent challenges in using a computer, which for non-blind users is mostly a visual experience.

And to not only become a proficient computer user but actually a developer, I think that's just super-neat.

I imagine this is where open source can really shine. Most open source software starts out it's life in an attempt to scratch your own itch. In the proprietary world if it doesn't look like there is enough money to be made, it won't get done (unless mandated by law).

But with open source you can crack open the source code and fix it yourself, with the help of other people with similar needs and interests.

The UCoC didn't come out of a vacuum. For a while Debian was really bad in that regard with the loudest, meanest members of the community poisoning the atmosphere. Free speech was overriding all other considerations. Nobody was ever punished for being particularly nasty. IMHO, it got to the point where it was really damaging the community experience and off-putting to newcomers.

When Ubuntu started, the leadership said - enough with that. We'll hold people up to higher standards. It seems to have worked!

I have two related concerns, both prefigured in an earlier post. I don't know the debian community, but I understand the reputation. And I know the Ubuntu community's reputation. And I can attest: when I've support, it was always ultimately a Ubunu person helping me or a group committed to Ubuntu. And in most every case the support was not only solid and methodically sound - it was also truly "supportive" and encouraging.

My second concern also comes out of an ignorance about Debian. I know Ubuntu is concerned about accessibility; I'm aware the accessibility team feels Lucid is no where near where it should be. But I know the commitment and appropriate priority is given to those that benefit from alternative input and output devices. I have no understanding of Debian's commitment to accessibility; but if it's waning, inconsistent, or outright oblivious, I would prefer my appliances to work on the platform that gives due diligence to accessibility.

I just discovered that Debian Squeeze was frozen a few days ago. See the full article here

In summary it will run the Linux 2.6.32 kernel and include (in its repos) Apache 2.2.16, PHP 5.3.2, MySQL 5.1.48, PostgreSQL 8.4.4 and Samba 3.4. It will also have interpreters and compilers for all common languages such as Python 2.6 and 3.1, Perl 5.10, GHC 6.12 and GCC 4.4.

I wonder how long it will be frozen for? Who knows? Some have reported that testing was frozen for 6 months prior to Lenny being released as stable! Hopefully it won't be that long!

64-bit support and our overhead: We don't yet know how 64-bit support will effect our workload. We're hoping it will mostly increase the aspects of the development process that can be batch automated. The real bottleneck is human labor, not computer resources. If that holds then there won't be a good reason to only support 64-bit on one platform.

Number of images: 50 appliances in 32-bit, 64-bit, Debian and Ubuntu versions. Then you have ISO, VM format, EC2 AMI, UEC, and Xen builds. That's about 1000 images! So yes we will soon be getting into crazy territory with regards to the number of images we are going to be pushing out. But we'll also be expanding how far we leverage cloud computing and automation in the build process so it shouldn't really make that much of a difference if we push out 100 or 1000 images.

Debian Lenny vs Squeeze: you make a good point with regards to the upcoming Squeeze release, but we're not inclined to go with Squeeze while it's unstable. There are still many release critical bugs I wouldn't want to expose users to. So we're going with Lenny for now but how far we go down that road depends on how much labor will go into "porting" over the appliances. Here we believe the main bottleneck will be testing, hence our reliance on the community to help pull through. Yes, the beta is an experiment. If it proves to be more than we can handle we may not release Debian based appliances for everything. Maybe only those applications that don't give us too much trouble.

Note that it doesn't look like squeeze is going to be released any time soon. According to the latest news from the release team they haven't even frozen the distribution yet. If I had to guess I'd say squeeze was at least 6 months away, maybe longer. So in the meantime, it would be nice to attract people from the Debian camp to TurnKey and help out with the first batch of Debian based appliances. I don't know if it will work but it would be nice to get the foot in the door with Lenny so by the time squeeze comes out we've expanded our community a bit with more people from the Debian community and make TurnKey Debian-based appliances a first class citizen.

There are positives and negatives for both distros. So I think the idea of having Debian and Ubuntu appliances concurrently and allowing community members to choose is a good option.

Having said that: Whilst I'm aware of many of the valid points you have raised above, I'm not sure whether there is or will be enough need or desire for the option of Debian vs Ubuntu for appliances that are almost identical in most ways. I guess that's part of the idea of this beta release? - To test that theory.

Here's an idea: What about doing Debian appliances as x86_64 only and Ubuntu appliances as x86? Its still probably not going to work for everyone but may provide enough differentiation to make it worth the effort? As you've mentioned you ultimately intend 64 bit support at some point, imagine the support load when that happens! Assuming you replicate all appliances as Debian & Ubuntu / 32 & 64 bit and increase the current appliance list to include additonal ones (say 10+ for arguments sake). That will require support for 200+ appliances (double that if you differentiate .iso & VM formats) which actually only fulfill 50+ appliance roles. Sounds like huge additional effort for minimal extra value!

Sorry to sound so down on an idea that I was so supportive of not that long ago! Don't get me wrong, I'm certainly not anti-Debian!

One other consideration: Whilst Squeeze isn't about to ship any day soon (and as you suggest Liraz, Debian are notorious for pushing back release dates, missing deadlines and ultimately just releasing "when ready") it is edging ever closer to release. I'd hate to see you guys put a huge amount of effort into releasing a new raft of Debian Lenny based appliances only to have Debian release Squeeze as 'stable' a few months later. Have you considered releasing the Debian appliance range based on Squeeze and retaining the 'beta' tag until it becomes 'stable'? (I'm guessing not as stability and consistency seem to be things you guys favour). I guess the irony of going that way is that once Squeeze is 'stable' it will actually have newer packages than those of the Ubuntu 10.04 based appliances - at least until 12.04 LTS.

If we're lucky they'll fix it before 10.04.1, but there's a real risk they won't fix it until the next LTS, maybe not even then.

I think reporting them is a good idea. Not that it probably makes a lot of difference, but I'd be happy to say that "this bug affects me too" if you let me know the bug numbers. I'm sure many other TKL users would be happy to do likewise.

As you say though it doesn't help us right now. Do you not think that some of the package maintainers/devs that it affects might be willing (perhaps even keen) to fix these issues prior to 10.04.1?

I don't know a lot about packaging but shouldn't it be fairly easy for them to fix? Sounds like a bit of a mess really, and a huge PITA for you guys. I guess its potentially fairly easy to fix each package, but once you fork it then no more updates eh? (unless you repackage each one - that sounds painful). Good luck with it. I for one would be interested to hear how you go, and what choices you make and why. Perhaps a blog post?

But in my book they're bugs and I've even talked to Alon about maybe reporting them on Launchpad. Not that it would do much good for the current release.

Right now, we're looking for workarounds that don't require us to fork too many packages, or break the package management system (too much).

So Canonical have also increased 'dependancies' for apps? I was aware that as of 10.04 they have made install 'recommends' the default behaviour when installing. But to also increase dependancies seems like a strange move to me. I can see that for many "average Joe" end users it may be quite helpful to install 'recommends' by default (as long as how to avoid this behaviour is well documented and fairly straight forward - which it is). But making additional 'dependancies' (that the application can function without - ie aren't really dependancies) as well just seems pure crazy to me. What is the point of that? Bloat for bloats sake??

Why TurnKey Core is 44% larger on Lucid: The increased size of the TurnKey Core beta on Lucid is something we're looking into and may soon be able to improve on but the main problem seems to be that the new version of Ubuntu simply specifies a more extensive set of dependencies, even when all we want is the same functionality that takes 44% less space in Debian Lenny and in Hardy, the previous version of Ubuntu.

This disappoints me because it should have been possible to make many of these "dependencies" (e.g., plymouth) optional (I.e., "recommends"), rather than mandatory. The proof is in the pudding: we have no intention of using plymouth for server appliances in TurnKey yet we're still forced to pull it in to satisfy dependencies. Unless we figure a way around this every Ubuntu based TurnKey appliance will include packages such as plymouth which we neither need nor use.

This is something that I doubt could happen in a Debian release for a number of reasons but mainly because Ubuntu's organizational structure allows for top-to-bottom mandates, even when they are very unpopular. There are pros and cons either way.

Responsiveness: the perceived speed difference between the Betas of Core on Lucid and Lenny may be primarily due to byobu, which is absent on the Lenny beta. This is a temporary difference. Based on the feedback we've received on performance issues and problems with nesting byobu will not be the default in future versions of TurnKey on Ubuntu either.

Update: there also seems to be a known issue with the framebuffer.

Minimal type CD not a viable base for a production system: Ubuntu's minimal CDs are pure network installers. If you dissect what's in there you'll discover that unlike a typical Live CD there is no root filesystem, just an initramfs. In the initramfs is a busybox based Linux environment just powerful enough to run debian-installer which then fetches everything else from the network.

It's utterly impractical to build TurnKey appliances using that as a basis. Keep in mind that TurnKey Core it itself built on TurnKey bootstrap which is already the smallest "legal" Ubuntu system that can boot and install new packages using apt-get.

I don't know about making it the default, but there's a good chance I'll be adding tmux to my personal toolset.

It's not just an issue of price, or even liberty. It's about security. An especially sensitive subject when you're dealing with NAT-traversing remote access. Proprietary, close-source software the community can't examine for backdoors doesn't sit well with me.

So if the benefits are strong enough I wouldn't rule out TurnKey running the required infrastructure. That's sort of what we're doing with the Hub...

but one thing many of these services offer which is not so easy to replicate in a free model is the unattended NAT/firewall tunneling they provide for non-tech users - and lazy ppl like me :). These solutions rely on a central man-in-the-middle server which connects the remote user to the system behind the NAT/firewall.

Personally I think this sort of big idea is admirable but is probably outside the scope of TKL.

OTOH there is probably freely available cross platform open source software that already provides most of the functionality you are looking for. Then perhaps a (yet to be designed) TKL gateway/firewall/VPN appliance that levarges DynDNS (or similar) to provide that same level of userfriendly NAT/Firewall tunneling. This could allow relativly non-tech users to set something similar up without the need to provide the third party man-in-the-middle server.

I have looked and looked for free open source solutions to the common problem of needing remote access software that clients can dl and it unistalls when it it done or one that has jump client that install on every computer in the network so that administrators can manage the network. Every single one cost thousands per user license. I think that it would be easy enough to develop our own live bootable OS that allows remote access into, WIN, MAC, Linux, and smartphones of all flavors that runs a LAMP stack for management. and I believe strongly that the turnkey core has the necessary starting point to develop it. I would love to have some people help me and put our heads together. if nothing else it will tear down the cost of the other companies software to a much lower cost. I think that if we sucessfully develop a OS/VM solution that hundreds of small and starting out companies would benefit greatly from such a software leap. Now obviously we will have to have mutliple platform gurus working on this project to develop a platform that supports all the above. SO please can we create a development site to start designing this incredible software. I would like to see it have the functionality of all 3 software titles. bomgar, logmein, and webex combined. so lets start dicussing features and concepts.

There is already an OpenVPN TKLPatch here that I imagine would be extremely useful (means you don't have to worry about that aspect). I haven't checked it out closely but perhaps thats getting close to what you're thinking?

As for other software you may want to include I'm not really sure. I'm happy to assist a bit and throw my 2c in but ATM I'm pretty snowed under so I probably won't be able to put in a lot of time (above and beyond my usual busy life, I'm also working on some TKL ideas).

The above patch apparently includes a Webmin OpenVPN module and the rest of the TKL hosted Webmin modules you can find here. (they should install into a TKL appliance with apt-get). If that still doesn't meet your needs then perhaps have a look on the Webmin site to see if they have a plugin (it'l be .tar.gz but we can worry about that later).

If your serious about taking this idea further then start another forum topic (I'd suggest in the General area).

Well if we can compile the already existing freely available cross Platform Open Source Software together with already existing gateway,firewall,VPN software into one single appliance then have it set to defualt boot to runlevel 2 so that it is manageable through webmin primarily as all tkl is and design the webmin to manage all of the above functions. Is that something that will be relatively medium to compile?

The general topic can be found here.

I used my google-fu and found the build system that Debian supports for custom live cds, here is a link to the home page:

http://live.debian.net/

It took me a while to figure out how to use it but I have a working live cd with tklpatch installed.  I am starting a thread in the general forum for some assistance on making it more inline with how the existing live cds work.

Here is a link to my forum post.

/forum/general/20110214/debian-squeeze-pre-alpha

Then using a bit of google-fu find yourself a tutorial for updating Lenny to Squeeze, test it on your clean install (documenting as you go). Once you have it working successfully put the instructions you used into the conf script of a tklpatch! Then package, test and post to a new ('general') forum topic.

I will give it a shot but I am a novice with linux and am even newer to TKLPatch.  When you guys get around to doing this will you just start with the current beta and run a dist upgrade or will you start with a clean install ISO?

Looks like Squeeze is now the stable release, any possibility of this release being updated to Squeeze?  If not, what is involved in making one of these core releases?

I prefer Debian on my servers and I love how easy it is to experiment by just taking this Debian core beta.