TurnKey Linux Virtual Appliance Library

TurnKey 13 out, TKLBAM 1.4 now backup/restores any Linux system

This is really two separate announcements rolled into one:

  1. TurnKey 13 - codenamed "satisfaction guaranteed or your money back!"

    The new release celebrates 5 years since TurnKey's launch. It's based on the latest version of Debian (7.2) and includes 1400 ready-to-use images: 330GB worth of 100% open source, guru integrated, Linux system goodness in 7 build types that are optimized and pre-tested for nearly any deployment scenario: bare metal, virtual machines and hypervisors of all kinds, "headless" private and public cloud deployments, etc.

    New apps in this release include OpenVPN, Observium and Tendenci.

    We hope this new release reinforces the explosion in active 24x7 production deployments (37,521 servers worldwide) we've seen since the previous 12.1 release, which added 64-bit support and the ability to rebuild any system from scratch using TKLDev, our new self-contained build appliance (AKA "the mothership").

    To visualize active deployments world wide, I ran the archive.turnkeylinux.org access logs through GeoIPCity and overlaid the GPS coordinates on this Google map (view full screen):

     

  2. TKLBAM 1.4 - codenamed "give me liberty or give me death!"

    Frees TKLBAM from its shackles so it can now backup files, databases and package management state without requiring TurnKey Linux, a TurnKey Hub account or even a network connection. Having those will improve the usage experience, but the new release does its best with what you give it.

    I've created a convenience script to help you install it in a few seconds on any Debian or Ubuntu derived system:

    URL=https://raw.github.com/turnkeylinux/tklbam/master/contrib/ez-apt-install.sh
    wget -O - -q $URL | PACKAGE=tklbam /bin/bash
    

    There's nothing preventing TKLBAM from working on non Debian/Ubuntu Linux systems as well, you just need to to install from source and disable APT integration with the --skip-packages option.

    Other highlights: support for PostgreSQL, MySQL views & triggers, and a major usability rehaul designed to make it easier to understand and control how everything works. Magic can be scary in a backup tool.

    Here's a TurnKey Hub screenshot I took testing TKLBAM on various versions of Ubuntu:

    Screenshot of TurnKey Hub backups

Announcement late? Blame my problem child

As those of you following TurnKey closely may have already noticed, the website was actually updated with the TurnKey 13.0 images a few weeks ago.

I was supposed to officially announce TurnKey 13's release around the same time but got greedy and decided to wrap up TKLBAM 1.4 first and announce them together.

TKLBAM 1.4 wasn't supposed to happen. That it did is the result of a spontaneous binge of passionate development I got sucked into after realizing how close I was to making it a lot more useful to a lot more people. From the release notes:

More people would find TKLBAM useful if:

  • If it worked on other Linux distributions (e.g., Debian and Ubuntu to begin with)

  • If users understood how it worked and realized they were in control. Magic is scary in a backup tool.

  • If it worked without the TurnKey Hub or better yet without needing a network connection at all.

  • If users realized that TKLBAM works with all the usual non-cloud storage back-ends such as the local filesystem, rsync, ftp, ssh, etc.

  • If users could more easily tell when something is wrong, diagnose the problem and fix it without having to go through TKLBAM's code or internals

  • If users could mix and match different parts of TKLBAM as required (e.g., the part that identifies system changes, the part that interfaces with Duplicity to incrementally update their encrypted backup archives, etc.)

  • If users could embed TKLBAM in their existing backup solutions

  • If users realized TKLBAM allowed them to backup different things at different frequencies (e.g., the database every hour, the code every day, the system every week)

    Monolithic all-or-nothing system-level backups are not the only way to go.

  • If it could help with broken migrations (e.g., restoring a backup from TurnKey Redmine 12 to TurnKey Redmine 13)

  • If it worked more robustly, tolerated failures, and with fewer bugs

So that's why the release announcement is late and Alon is slightly pissed off but I'm hoping the end result makes up for it.

TurnKey 13: from 0.5GB to 330GB in 5 years

Big things have small beginnings. We launched TurnKey Linux five years ago in 2008 as a cool side project that took up 0.5GB on SourceForge and distributed 3 installable Live CD images of LAMP stack, Drupal and Joomla.

5 years later the project has ballooned to over 330GB spanning 1400 images: 100 apps, 7 build types, in both 64-bit and 32-bit versions. So now we're getting upset emails from SourceForge asking if the project really needs to take up so much disk space.

Yes, and sorry about that. For what it's worth, realizing TurnKey may eventually outgrow SourceForge is part of the reason we created our own independent mirror network (well, that and rsync/ftp access). Sourceforge is great, but just in case...

93,555 lines of code in 177 git repos

In terms of development, I recently collected stats on the 177 git repositories that make up the app library, self-contained build system, and a variety of custom components (e.g., TKLBAM, the TurnKey Hub).

It turns out over the years we've written about 93,555 lines of code just for TurnKey, most of it in Python and shell script. Check it out:

Late but open (and hopefully worth it)

TurnKey 13 came out a few months later than we originally planned. By now we have a pretty good handle on what it takes to push out a release so the main reason for the delay was that we kept moving the goal posts.

In a nutshell, we decided it was more important for the next major TurnKey release to be open than it was to come out early.

The main disadvantage was that Debian 7 ("Wheezy") had come out in the meantime and TurnKey 12 was based on Debian 6 ("Squeeze"). On the other hand Debian 6 would be supported for another year and since TurnKey is just Debian under the hood nothing prevented impatient users who wanted to upgrade the base operating system to Debian 7 to go through the usual automated and relatively painless Debian upgrade procedure.

So we first finished work on TKLDev, put it through the trenches with the TurnKey 12.1 maintenance release, and moved the project's development infrastructure to GitHub where all development could happen out in the open.

We hoped to see a steady increase in future open source collaboration on TurnKey's development and so far so good. I don't expect the sea to part as it takes more than just the right tools & infrastructure to really make an open source project successful. It takes community and community building takes time. TurnKey needs to win over contributors one by one.

Alon called TurnKey 13.0 "a community effort" which I think in all honesty may have been a bit premature, but we are seeing the blessed beginnings of the process in the form of a steadily growing stream of much appreciated community contributions. Not just new prototype TurnKey apps and code submissions but also more bug reports, feature requests and wiki edits.

And when word gets out on just how fun and easy it is to roll your own Linux distribution I think we'll see more of that too. Remember, with TKLDev, rolling your own Debian based Linux distribution is as easy as running make:

root@tkldev ~$ cd awesomenix
/turnkey/fab/products/turnkey/awesomenix
root@tkldev turnkey/awesomenix$ make

You don't even have to use TKLDev to build TurnKey apps or use any TurnKey packages or components. You can build anything you want!

Sadly, I've gotten into the nasty habit of prepending TKL - the TurnKey initials - to all the TurnKey related stuff I develop but under the hood the system is about as general purpose as it can get. It's also pretty well designed and easy to use, if I don't (cough) say so myself.

I'll be delighted if you use TKLDev to help us improve TurnKey but everyone is more than welcome to use it for other things as well.

3 new TurnKey apps - OpenVPN, Tendenci and Observium

  • OpenVPN: a full-featured open source SSL VPN solution that accommodates a wide range of configurations, including remote access, site-to-site VPNs, Wi-Fi security, and more.

    Matt Ayers from Amazon asked us to consider including an OpenVPN appliance in the next release and Alon blew it out of the park with the integration for this one.

    The new TurnKey OpenVPN is actually a 3 for 1 - TurnKey's setup process asks whether you want OpenVPN in client, server or gateway mode and sets things up accordingly.

    My favourite feature is the one that allows the admin to create self destructing URLs with scannable QRcodes that makes setting up client OpenVPN profiles on mobiles a breeze. That's pretty cool.

  • Tendenci: a content management system built specifically for NPOs (Non Profit Organizations).

    Upstream's Jenny Qian did such an excellent job developing the new TurnKey app that we accepted it into the library with only a few tiny modifications.

    This is the first time an upstream project has used TKLDev to roll their own TurnKey app. It would be awesome to see more of this happening and we'll be happy to aid any similar efforts in this vain any way we can.

  • Observium: a really cool autodiscovering SNMP based network monitoring platform.

    The new TurnKey app is based on a prototype developed by Eric Young, who also developed a few other prototype apps which we plan on welcoming into the library as soon as we work out the kinks. Awesome work Eric!

Special thanks

Contributing developers:

Extra special thanks

  • Alon's wife Hilla: for putting up with too many late work sessions.
  • Liraz's girlfriend Shir: for putting up with such a difficult specimen (in general).
You can get future posts delivered by email or good old-fashioned RSS.
TurnKey also has a presence on Google+, Twitter and Facebook.

Comments

As usual a stellar performance guys!

You guys deploy at an amazing pace. Just keeping up with you is a full time job :-)

Nice work...as usual. Impressive global TKLX map!

Cheers,

Tim (Managing Director - OnePressTech)

Liraz Siri's picture

Must... get... sleep...

Thanks for the support Tim! We whip ourselves into a frenzy right before making big announcements. I think I may have gotten a bit too greedy this time around because the secret sauce I'm running on at the moment is sleep deprivation. Will go get some much needed rest as soon as I finish updating the copy of the TKLBAM FAQ on the website.

Website Problem

There is a problem on Chrome 31 and IE 11 on the Content Management category. When I press next to see the rest of the build that are available, the circle spins forever and does not go to the next page. This problem is only for the Content Management Category, the rest of the categories work just fine.

Liraz Siri's picture

Temporary glitch

Thanks for reporting this, but I can't reproduce it anymore so I think it was a temporary glitch. I made a handful of changes to the site templates right before the announcement and cleared the site's cache. The load spiked up something awful.

The problem with the spinning

The problem with the spinning circle still persists, and now the Web development category is affected as well. I think it would be better to change the system of displaying the appliances as this bug is very annoying.

Jeremy's picture

I can confirm this behaviour!

Hey Liraz,

I can confirm this behaviour! To reproduce:

  1. Go to http://www.turnkeylinux.org/
  2. Scroll down to the bottom of the appliance list and click '2' (i.e. page 2 of results), '3', 'next >' or 'last >'
  3. Little blue spinning circle appears to the right of it...
  4. And just sits there spinning...

Currently I have the TKL home page loaded on 'Content Management and have all 4 little blue circles spinning and they have been going for a good 5-10 mins!

I have tested both Chrome (v31.0.1650.57 m) & Firefox (v25.0.1) on Win 7 x64 and the behaviour is consistent...

The page navigation links only seem to work when you are on the all appliances tab.

[Edit] It looks like this has been happening for a while. Here is the bug lodged by Alon.

Eric (tssgery)'s picture

I don't see it with my Linux desktop

I don;t know if this will helo isolate the problem but things seem to work fine with both firefox (25.0.1) and chrome (30.0.1599.114) on my Ubuntu 13.10 desktop


Jeremy's picture

Hmmm interesting...

So perhaps it's a Windows thing then...!?

Jeremy's picture

Really weird...

Now I have Debian up and running on my laptop (again) I thought I retest this and it was weird...

When I initially tested it it was working fine (for the full list - confirming the theory that it was something to do with Windows) but then when I went into one of the sub headings (e.g. Content management) the (bad) behaviour was there... So I started editing my post above to add that step in, but while I was doing that I restested a few times, and guess what! It is now working fine... However, the browser tab that I first tested it in (which is still open) is still displaying the (bad) behaviour (i.e. the little blue spinning circle is still sitting there spinning...) Strangeness...!

Liraz Siri's picture

Worse thing that can happen if the connection drops

Hi Carl, thanks for the warning but case in point, the worst thing that could happen if the connection drops is that TKLBAM will fail to install. The convenience script doesn't execute any dangerous commands (e.g., rm) so the worst case edge case is rather mild.

Also, I'm expecting anyone who really cares about this sort of thing to also understand that they don't have to pipe the script directly through to the shell. It's safer to download, take a look at the script, chmod +x and then execute it. But then it wouldn't be a neat oneliner.

Great!

Congratulations!

Now I'm 90% satisfied user. The 10% left is because I've got to *work* to update all my instances in production. :D

Thanks a lot team, Turnkey is a great product!

 

 

Liraz Siri's picture

I know what that's like

Thanks Hernan! FWIW, I have a few Debian Squeeze to Wheezy upgrades scheduled myself. Previously I would backup the server, ask apt-get to do a dist-upgrade and hope for the best but I'm hoping this time around to see if I can use the new version of TKLBAM to move my Squeeze configurations to Wheezy more efficiently.

John Carver's picture

Congratulations!

And thanks for posting the user map.  I was under the impression that I was the only TurnKey user in Iowa but I see that is not the case.  If there are any TurnKey users in the I-380 corridor who'd like to meet for coffee, send me a note.  Maybe we can start a user group.

Information is free, knowledge is acquired, but wisdom is earned.

Liraz Siri's picture

Thanks John! Speaking of user groups...

Thanks John. If the map I whipped up helps you make a friendly new acquaitance in your area that would awesome!

Speaking of TurnKey user groups Alon and I sat down for our traditional post-release chat and decided encouraging (and possible coordinating) TurnKey user groups would be a great idea. We need more feedback from users and we need to do a better job of engaging with the community. Alon and I are too in love with the technical side of stuff to be very good at that sort of thing so we've decided it's time to look for full time help.

k0nsl's picture

Thanks !

Thanks Mr. Siri for the update. Turnkey is very useful and we have most of the images available to our clients (free hosting).

All the best.

I'm a programmer, server administrator, designer, spokesman and factotum of a wide-variety of web pages.

Liraz Siri's picture

Thanks k0nsl

Thanks for offering TurnKey to your clients. If you have more feedback on how we can make things better, let us know. We're always on the lookout for new ways to improve TurnKey.

Fantastic

Sincerest congratulations and an enormous thank you! We are neck deep in TKLDEV. A revolution.


Sweet nibblets...

So I got the email about this post and I acutally did a little dance... So excited for v13! Love the news about the TKLBAM. That is so sweet news. Can't wait to dive into the TKLDev just so much awesomeness...

Also nice map of the TKL installs, I think I spotted one of my installs on the map, not too many people got TKL in Mechanicsburg PA. Thanks again!

Happy TurnKeying... ":0)

 

-Ken

64 aws image?

just curious when a 64 bit turnkey box might be released - keep up the good work & thank you

John Carver's picture

64 bit images have been

64 bit images have been available since version 12.1.  I'm not an Amazon user (yet) but I can't imagine they aren't available there.

Information is free, knowledge is acquired, but wisdom is earned.

You're absolutely correct - I

You're absolutely correct - I just didn't see them on the TKL site - if you go in to the amazon load a new ami area they're all there

Thanks!

 

Liraz Siri's picture

The download links defaults to the 64-bit images

Since 12.1 the site download link has defaulted to the 64-bit images. The 32-bit images are still there, on sourceforge and on the mirror network but they're deprecated and may not make it into the next release. This depends on the demand we get from the community though. If enough people really want 32-bit builds we'll continue building them. It just doubles the storage requirements for our mirrors...

Jeremy's picture

I would still like to see 32 bits into the future...

I know that most hardware these days supports 64 bit but for virtualisation I think you'll find that some people struggle to get 64 bit images working in VirtualBox (for example...) I know others have posted about having issues launching 64 bit images (and have resolved it by installing the 32 bit ones). I too have encountered this (in Win 7 x64) despite the fact that I have all the virtualisation options turned on in my BIOS. It may be some setting in VirtualBox (or perhaps a bug?) that also needs to be correctly set but I just haven't had the time to play around with it (nor tested it in Debian)...

Just my 2c! :)

Thanks for the release

Great job, tkl is great, tklbam rocks thanks.

Liraz Siri's picture

Thanks for trying TKLBAM. Feedback welcome...

TKLBAM is my baby, so any feedback you can provide that could help me improve the usage experience would be greatly appreciated. Cheers!

Thank you - you guys are legendary.

Thank you Turnkey Linux Team. Your great and untiring efforts are much appreciated. I love Turnkey Linux because it's Debian !!!

 

More power to "Turnkey Linux Team"

Doy from "Down Under"

Liraz Siri's picture

Debian rocks 37,000 times over

Nearly 37,000 packages in Wheezy. All supported with backported security updates that are so reliable you can install them automatically. What's not to love?

Jeremy's picture

Great work guys!

Congrats on the 'official' release of v13.0, although I must admit I have been using them for some time! :)

And TKLBAM v1.4 sounds fantastic. I'm yet to play with it much but I love the fact that you have 'unchained' it! Having it default to S3 is handy but sometimes it's nice to use local storage (or other stuff) and making that scenario more user friendly is awesome!

All in all it seems to me that TKL is maturing nicely! :)

PS I too love the user map! It seems like there are other TKL users in Tasmaina too! Yay! :)

How to upgrade form last version

Hi guys,

 

I'm running the lamp appliance, version 12 on debian 6, within esxi 5.1.  Any suggestions for how to upgrade turnkey core?  I couldn't locate anything in webmin.  Any help is much appreciated.

 

BTW, congrats and great job!

 

-d
 

How to upgrade form last version

Ok,  I read the docs and it seems like I have to backup to tklbam, install the new version in a new VM and restore from backup.  Does that sound about right?

 

Thanks again,

 

-d

Jeremy's picture

Yes that's the best way

Also if you don't want to use AWS S3 storage then I think you should be able to install the new version of TKLBAM (v1.4) which should easily allow you to backup somewhere else (e.g. another local server, etc).

OSQA VM

Hi guys,

congrats for the 13.0 release and also for the new TKLBAM version!

I was really happy to see that I won't need cloud storage any more just to fetch the backup-profile for the given VM in TKLBAM. It is also great news, that it now supports backing up PostgreSQL databases as well.

I have recently wanted to install the OSQA VM, but the newest version available on the website is 12.1, which is not fully compatible with XenServer. Do you plan to make the 13.0 version available?

Best,
Endre

Jeremy's picture

There was an issue building the v13.0 OSQA appliance

SO unforunately it is currently blocked. See the bug report here. Basically OSQA depends on the legacy version of Django which is not available in Wheezy... :( 

So until they update to support the version of Django in the Debian repos it wil remain blocked...

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)