TKLBAM: a new kind of smart backup/restore system that just works

Drum roll please...

Today, I'm proud to officially unveil TKLBAM (AKA TurnKey Linux Backup and Migration): the easiest, most powerful system-level backup anyone has ever seen. Skeptical? I would be too. But if you read all the way through you'll see I'm not exaggerating and I have the screencast to prove it. Aha!

This was the missing piece of the puzzle that has been holding up the Ubuntu Lucid based release batch. You'll soon understand why and hopefully agree it was worth the wait.

We set out to design the ideal backup system

Imagine the ideal backup system. That's what we did.

Pain free

A fully automated backup and restore system with no pain. That you wouldn't need to configure. That just magically knows what to backup and, just as importantly, what NOT to backup, to create super efficient, encrypted backups of changes to files, databases, package management state, even users and groups.

Migrate anywhere

An automated backup/restore system so powerful it would double as a migration mechanism to move or copy fully working systems anywhere in minutes instead of hours or days of error prone, frustrating manual labor.

It would be so easy you would, shockingly enough, actually test your backups. No more excuses. As frequently as you know you should be, avoiding unpleasant surprises at the worst possible timing.

One turn-key tool, simple and generic enough that you could just as easily use it to migrate a system:

  • from Ubuntu Hardy to Ubuntu Lucid (get it now?)
  • from a local deployment, to a cloud server
  • from a cloud server to any VPS
  • from a virtual machine to bare metal
  • from Ubuntu to Debian
  • from 32-bit to 64-bit

System smart

Of course, you can't do that with a conventional backup. It's too dumb. You need a vertically integrated backup that has system level awareness. That knows, for example, which configuration files you changed and which you didn't touch since installation. That can leverage the package management system to get appropriate versions of system binaries from package repositories instead of wasting backup space.

This backup tool would be smart enough to protect you from all the small paper-cuts that conspire to make restoring an ad-hoc backup such a nightmare. It would transparently handle technical stuff you'd rather not think about like fixing ownership and permission issues in the restored filesystem after merging users and groups from the backed up system.

Ninja secure, dummy proof

It would be a tool you could trust to always encrypt your data. But it would still allow you to choose how much convenience you're willing to trade off for security.

If data stealing ninjas keep you up at night, you could enable strong cryptographic passphrase protection for your encryption key that includes special countermeasures against dictionary attacks. But since your backup's worst enemy is probably staring you in the mirror, it would need to allow you to create an escrow key to store in a safe place in case you ever forget your super-duper passphrase.

On the other hand, nobody wants excessive security measures forced down their throats when they don't need them and in that case, the ideal tool would be designed to optimize for convenience. Your data would still be encrypted, but the key management stuff would happen transparently.

Ultra data durability

By default, your AES encrypted backup volumes would be uploaded to inexpensive, ultra-durable cloud storage designed to provide %99.999999999 durability. To put 11 nines of reliability in perspective, if you stored 10,000 backup volumes you could expect to lose a single volume once every 10 million years.

For maximum network performance, you would be routed automatically to the cloud storage datacenter closest to you.

Open source goodness

Naturally, the ideal backup system would be open source. You don't have to care about free software ideology to appreciate the advantages. As far as I'm concerned any code running on my servers doing something as critical as encrypted backups should be available for peer review and modification. No proprietary secret sauce. No pacts with a cloudy devil that expects you to give away your freedom, nay worse, your data, in exchange for a little bit of vendor-lock-in-flavored convenience.

Tall order huh?

All of this and more is what we set out to accomplish with TKLBAM. But this is not our wild eyed vision for a future backup system. We took our ideal and we made it work. In fact, we've been experimenting with increasingly sophisticated prototypes for a few months now, privately eating our own dog food, working out the kinks. This stuff is complex so there may be a few rough spots left, but the foundation should be stable by now.

Seeing is believing: a simple usage example

We have two installations of TurnKey Drupal6:

  1. Alpha, a virtual machine on my local laptop. I've been using it to develop the TurnKey Linux web site.
  2. Beta, an EC2 instance I just launched from the TurnKey Hub.

In the new TurnKey Linux 11.0 appliances, TKLBAM comes pre-installed. With older versions you'll need to install it first:

apt-get update
apt-get install tklbam webmin-tklbam

You'll also need to link TKLBAM to your TurnKey Hub account by providing the API-KEY. You can do that via the new Webmin module, or on the command line:

tklbam-init QPINK3GD7HHT3A

I now log into Alpha's command line as root (e.g., via the console, SSH or web shell) and do the following:

tklbam-backup

It's that simple. Unless you want to change defaults, no arguments or additional configuration required.

When the backup is done a new backup record will show up in my Hub account:

To restore I log into Beta and do this:

tklbam-restore 1

That's it! To see it in action watch the video below or better yet log into your TurnKey Hub account and try it for yourself.

Quick screencast (2 minutes)

Best viewed full-screen. Having problems with playback? Try the YouTube version.

The screencast shows TKLBAM command line usage, but users who dislike the command line can now do everything from the comfort of their web browser, thanks to the new Webmin module.

Getting started

TKLBAM's front-end interface is provided by the TurnKey Hub, an Amazon-powered cloud backup and server deployment web service currently in private beta.

If you don't have a Hub account already, request an invitation. We'll do our best to grant them as fast as we can scale capacity on a first come, first served basis. Update: currently we're doing ok in terms of capacity so we're granting invitation requests within the hour.

To get started log into your Hub account and follow the basic usage instructions. For more detail, see the documentation.

Feel free to ask any questions in the comments below. But you'll probably want to check with the FAQ first to see if they've already been answered.

Upcoming features

  • PostgreSQL support: PostgreSQL support is in development but currently only MySQL is supported. That means TKLBAM doesn't yet work on the three PostgreSQL based TurnKey appliances (PostgreSQL, LAPP, and OpenBravo).
  • Built-in integration: TKLBAM will be included by default in all future versions of TurnKey appliances. In the future when you launch a cloud server from the Hub it will be ready for action immediately. No installation or initialization necessary.
  • Webmin integration: we realize not everyone is comfortable with the command line, so we're going to look into developing a custom webmin module for TKLBAM. Update: we've added the new TKLBAM webmin module to the 11.0 RC images based on Lucid. In older images, the webmin-tklbam package can also be installed via the package manager.

Special salute to the TurnKey community

First, many thanks to the brave souls who tested TKLBAM and provided feedback even before we officially announced it. Remember, with enough eyeballs all bugs are shallow, so if you come across anything else, don't rely on someone else to report it. Speak up!

Also, as usual during a development cycle we haven't been able to spend as much time on the community forums as we'd like. Many thanks to everyone who helped keep the community alive and kicking in our relative absence.

Remember, if the TurnKey community has helped you, try to pay it forward when you can by helping others.

Finally, I'd like to give extra special thanks to three key individuals that have gone above and beyond in their contributions to the community.

By alphabetical order:

  • Adrian Moya: for developing appliances that rival some of our best work.
  • Basil Kurian: for storming through appliance development at a rate I can barely keep up with.
  • JedMeister: for continuing to lead as our most helpful and tireless community member for nearly a year and a half now. This guy is a frigging one man support army.

Also special thanks to Bob Marley, the legend who's been inspiring us as of late to keep jamming till the sun was shining. :)

Final thoughts

TKLBAM is a major milestone for TurnKey. We're very excited to finally unveil it to the world. It's actually been a not-so-secret part of our vision from the start. A chance to show how TurnKey can innovate beyond just bundling off the shelf components.

With TKLBAM out of the way we can now focus on pushing out the next release batch of Lucid based appliances. Thanks to the amazing work done by our star TKLPatch developers, we'll be able to significantly expand our library so by the next release we'll be showcasing even more of the world's best open source software. Stir It Up!

Comments

Dan Robertson's picture

Great job!

Congratulations !

Adrian Moya's picture

This is incredible news for TKL Community! Thanks for the hard work on that, it's really a wonderful backup system. It gives a great amount of added value to TKL. 

Jomy Muttathil's picture

Very Impressive!

Is there be an option for automated incremental backups?

I like how it is not just for backups but can be used for migration as well.  Very clever.

 

This is Huge!

Liraz Siri's picture

To enable daily incremental backups you just do this:

chmod +x /etc/cron.daily/tklbam-backup

The FAQ's usage section explains the difference between full and incremental backups and how to configure how frequently a full backup happens (in between full backups all backups are incremental).

Liraz Siri's picture

We've been working on this harder than we've worked on any other TurnKey feature. It feels wonderful to be able to finally share TKLBAM with the world and it means a lot to us when people get as excited about it as we are!

If I understand the documentation correctly, TKLBAM may be an essential part of our data management within the next week. I'll be rereading a few times.

Again, I realize how important this software is to the community and have only the sincerest congratulations, thanks, and admiration.

jgab's picture

Congratulations and thanks to the development team!

I cannot put anything out in the cloud - only locally - will this happen anytime soon?

Regards

Richard

Liraz Siri's picture

You just won't get to enjoy the streamlined usability enabled by the Hub, so you'll have to manage backups, keys and authentication credentials by hand.

See the FAQ for details.

Anyhow, there is no schedule yet for streamlined usability in a private cloud type usage scenario. We'll revisit this after the Lucid release. We have limited resources, so we have to pick our battles carefully.

Jomy Muttathil's picture

The logical next step would be a Turnkey Linux TKLBAM Appliance.

A TKL appliance for backing up and migrating other TKL appliances.

Everything comes full circle.

Seconded !


Jeremy Davis's picture

I go away for a couple of days and don't check my emails and THIS happens!

But seriously - great work guys! TKLBAM looks impresive and builds nicely on the Hub. I think the name is fantastic too. TKLBAM has such a nice ring to it!

Also thanks for the kudos guys. I do it all for the love! I really enjoy feeling like part of the team and I feel honoured that I can be a part of such a great project! The community really seems to be growing and maturing of late which is awesome to see. Like you, I have been amazed with the TKLPatches that Basil & Adrian have been putting together.

Once again a worthy addition to the TKL brand!

Liraz Siri's picture

You disappeared for a few days, so naturally Alon and I decided we had to do something immediately to lure you back. My idea was to fire up the Jed-Signal. Then Alon said, "I have a better idea, let's release that backup and migration system we've been working on for the past year". "You'll think it'll work?" I asked. "Let's just pray it does. Now quickly, to the TurnKey-mobile!"

I'm just glad it worked...

Mark McIntosh's picture

This is a great peice of work Thanks!!! I have a few questions as this is my first time working with a Turnkey linux appliance. I normally just use an ubuntu image (generally ebs root) from scratch. I love the idea of being able to backup and migrate this easily but have a few questions. Does it have to be a turnkey appliance to work ??? Can I add an existing dev server to test with ?? I saw some mentions above of the possiblity of a private cloud version and add my vote for that kind of feature.

Once again thanks for the great work.

Mark

Liraz Siri's picture

1) In automatic mode, you can use TKLBAM anywhere (e.g., private cloud) so long as you have network access to the Internet and can reach Amazon S3 and the TurnKey Hub. If your private cloud isn't firewalled off from the Internet it will work. In manual mode, you don't even need access to Amazon S3. You can store your backups on the local network. It's just not as easy to use.

2) TKLBAM depends on a few things that are currently TurnKey specific (e.g., a fixed starting point after installation / AKA the "appliance" concept). This isn't just some arbitrary limitation, but part of what powers the zero-configuration magic. Behind the scenes we've hand-crafted the default backup configurations for each appliance in the TurnKey library. In other words, we can make a very good guess at where you data is stored if you're using TurnKey File Server (for example) because we basically set it up for you.

Generic Ubuntu/Debian takes a different approach where the installation process is more general purpose. The installer asks you what kind of system you want, what additional packages to install, etc. So... no fixed starting point at installation.

But remember that under the hood TurnKey is just a series of Ubuntu (and soon Debian also) based systems optimized for a particular usage scenario. When you use TurnKey you are not making the choice not to use Ubuntu. You are making a choice to use an Ubuntu system we pre-integrated for you.

jgab's picture

Hi, you can find an interesting article about Turnkey Linux on Linux Magazine:

http://www.linux-mag.com/id/7857?hq_e=el&hq_m=1071524&hq_l=3&hq_v=fb6a4a01db

Good!

Alon Swartz's picture

Thanks for the link jgab, you actually beat me to the punch.

I'd like thank all the online publications and writers who have covered TKLBAM's announcement, and helping us get the word out. For your reading pleasure, here is the list of articles I came across, listed in order of publication date.

 
Let me know if I missed any.
TonyRLA's picture

Hi everyone.  I just learned about the work you guys are doing and am in awe.  I am also both excited and relieved to have found that you have developped exactly the product I'm looking for - I think.  My question is simple:  How do I get my existing deployments into TKL?  I've spent a few hours reading through stuff and haven't seen a definitive answer (some getting started docs are needed, and in the spirit of Open Source I'm willing to pitch-in if help is needed).

It seems like I would use TKLBAM to import the file structures lock stock and barrel. Is that correct?

I have 2 production instances running at Linode, both on Ubuntu 10.04.  One is a WordpressMU install, the other is a custom app built on django. 

So would I start with a couple of ubuntu instances on the hub and go from there?

And sorry if this question is already clearly answered somewhere, but I've looked thoroughly and not found the answer.  If the answer is lying around soemwhere, I'd humbly suggest putting it soemwhere front and center where it's easy to find.

Thanks!


Jeremy Davis's picture

TKLBAM can only be used for migration between TKL appliances (eg from the Hardy based v2009.10-2 to the new v11.0). That means that you cannot do what you are hoping to.

I would suggest that you install TKL (to wherever) and rsync from your existing server to the relevant TKL instance (WordPress and Django I would imagine). Hopefully you'll be lucky and it all works smoothly. Once you have it all transferred accross and working sweetly you can then use TKLBAM to back it up.

As for docs, yes you are right. We need more and better! Adrian has made a start on tidying that up here on the Dev Wiki. Please feel free to put some rough drafts, links to forum threads you find useful and/or any info you think should be in the official docs (you will need to register/login seperately for the Dev Wiki).

TonyRLA's picture

OK, so I should just set-up a local instance of TKL Core and rsync with fingers crossed? 

One concern is that there have been some server tweaks done on both installs but especially the django app so I'd like to just pick up the whole system and plunk it down if I could.

I guess I've just recommended another product for you guys...

I know VMWare has a client that does that, it's called the configurator or something like that.


Jeremy Davis's picture

I would use TKL Django and WordPress appliances respectively (rather than Core). That way you will be able to take full advantage if TKLBAM for backups (TKLBAM will only backup what is needed - if you use Core you will have larger backups unneccesarily).

As TKL v11.x is based on Ubuntu 10.04, assuming that your current appliances are using default locations for content and settings, you should only have to rsync content and settings files and dump/import the MySQL database(s). If your current deployments are set up 'properly' then most, if not all of your tweaks should also transfer across no worries.

I am happy to try to assist you (but no promises about final outcomes) with more fine grained help but it's probably better to start a new forum thread (I'll see it, but feel free to post a link to your new thread here too).

sbscherer's picture

I think I overlooked an important detail about the hub.  Does the hub allow a custom appliance, lets say built on a Core/LAMP/LAPP appliance, to be properly backed up to the hub regardless of the type of application running on top?

Jeremy Davis's picture

The Hub will assume that it is the base appliance that you used and will backup user editable locations and will create a reinstall list of installed packages. Assuming you used the standard repos included with TKL and haven't put anything in 'naughty places' then it will all go according to plan.

However if you have additional repos or manually installed .debs then I won't be able to install them. Also you may wish to do manual overide to include specific paths that are not included by default. See Liraz's post here for more detail.

Best way to go is to test it out and have a bit of a play.

TonyRLA's picture

I started a new topic here: http://bit.ly/gMnti3

I proposed one method I thunk-up, and posted some other possible resources I found. 

I'd love to hear your thoughts, and am grateful for your offer to help. I'll take you up on it, and I totally absolve you of any responsibility for results.

Thanks again.


Des O'Toole's picture

First off, contratulations on a great feature. I've been following your work here for some time but I've only just got around to trying it out; it's an impressive feature that you've created, well done!

One slight niggle though, the incremental backups are great, but the restore feature seems to download a full restore each time. I was thinking of using tklbm to keep 2 machines in sync (plus a remote backup), so a full download each time is a bit excessive. It occurs to me that if you kept a local copy of the last restore then you could use rsync to do an incremental restore (assuming the hub is capable of doing this of course). What do you think?

Thanks for all the great work.

Des

Liraz Siri's picture

Thanks for the feedback Des. I'll have to think about this a bit more but this could be a good idea, even though TKLBAM was never designed for this specific usage scenario. It's impractical to satisfy all usage scenarios with a single, well designed tool, but if with a modest change we could provide system-level syncing capabilities that would be worthwhile.

Note however the implementation would be a bit more involved than you seem to suspect. TKLBAM stores backups on S3, which doesn't support the rsync protocol natively. Also, we use Duplicity on the back-end and Duplicity doesn't have support for this type of caching, though it may be possible to get around that without diverging more from the main Duplicity branch.

Mark Vossen's picture

Hi

I noticed an issue when trying to restore folders that contain spaces or accents.
The command line tklbam-restore help syntax explains:
 --limits="LIMIT-1 .. LIMIT-N"     Restore filesystem or database limitations
      LIMIT := -?( /path/to/include/or/exclude | mysql:database[/table] )
These parameters seems to work fine, unless the foldernames used contain special characters such as spaces or accents.
Restoring files from a folder that contains spaces in its foldername doesn't seem to work when specifying that foldername as a restore-point. TKLBAM doesn't restore any files. The logfile /var/log/tklbam-restore contains no warnings.
 
To simulate this problem, create a file in a folder called 'New Folder' in /data
Put a file in \data\New Folder and run a backup.
Next, try to restore that folder.
I tried this by entering:
tklbam-restore --skip-database --skip-packages --limits='/data/New Folder/' 8
where 8 is the ID of my backup.
No restore takes place. (also tested with extra \ tklbam-restore --skip-database --skip-packages --limits='/data/New\ Folder/' 8)
 
tklbam-restore --skip-database --skip-packages --limits='/data/New Folder/' 8
does work.
In the understanding that there is no option to restore to an alternate path, this means that a sysadmin neds to overwrite the entire /data folder to restore a single file in /data/New Folder ?
Please advise
Kind regards
Mark Vossen
 
 
Mike Gifford's picture

It's great that you've built and documented how to add the backups. I don't yet understand how to remove them? I'd like to keep a nice rotation, but want to make sure I'm not backing things up on Amazon forever if I don't need to.

Jeremy Davis's picture

If you go into the backups section and open the relevant server it can be set there (default is unlimited).

Mike Gifford's picture

What happens when you set a backup limit? What happens when you hit or cross that line?  Are old backups deleted automaticly?

Jeremy Davis's picture

I've got mine set up to keep 2 backups. That means that once a third full backup is made the oldest one is automatically deleted. Note that this setting is per server (so they can be different).

Mike Gifford's picture

I like having backup rotations where if a problem isn't noticed for a few days it's possible to go back and restore stuff.  Ideally it would be possible to go back even a month.

I'd like to see:

  • daily incremental backups
  • full weekly backup
  • full monthly backup

I'm used to backing up with rsync, just focusing on /etc the drupal directory & mysql dumps

Jeremy Davis's picture

I know it's not exactly what you are after but it's close.

Enable daily incremental backups. Set full backups for 2 weekly, and keep 2 full backups.

Then you get relatively short incremental chains and minimal storage usage, plus up to a month old full backup.

Liraz Siri's picture

Yes, if you set a limit the oldest backup chain is deleted by the Hub automatically every time you create a new chain that exceeds the limit.

Mike Gifford's picture

There are a bunch of common cache files in Drupal that i'd rather not back up.  Does:

echo -mysql:*/sessions >> /etc/tklbam/overrides

or

echo -mysql:*/cache >> /etc/tklbam/overrides
echo -mysql:*/cache_* >> /etc/tklbam/overrides

work to override generic database tables?

Jeremy Davis's picture

I always just use AWS (cause it's so easy and cheap - also it's good to have an offsite backup IMO). But I suspect your suspicion is correct. But to be on the safe side, rather than copy paste the key, I'd be inclined to copy the keyfile out using SFTP (so the full formatting of the file is preserved).

Jeremy Davis's picture

I don't bother with the extra layer of encrytion. I thought you were trying to run a backup to a local location (rather than S3/AWS - Amazon Web Services). Bottom line is - sorry I have no idea. I'd be inclined to just test it and see what happens. If you try restoring to a fresh AWS appliance it will be really quick (and cheap too) then you'll know.

Jeremy Davis's picture

But in my experience TKLBAM works best on a clean instance. Perhaps try it and see how it goes. If it works, great. Otherwise start up a new clean TKL VM and restore to that.

Jeremy Davis's picture

This assumes that your starting point is a cloud server.

  1. Back it up with TKLBAM.
  2. Temporarily lock your cloud server (if it's a static site that's easy, if a more dynamic site - lock forums/comments etc).
  3. Deploy a fresh local vm and restore your backup to it.
  4. Make the changes you wish to make on your local VM, test etc.
  5. Make a new TKLBAM backup of it.
  6. Restore your new backup to your original server (or even create a fresh instance) and unlock it again (you'll need to reconfigure your DNS etc if you start a fresh instance).

Next time you want to make adjustments or test a new feature - whatever, just repeat the steps...

You could use git or something instead but I like TKLBAM because I can easily move customised servers between different VM solutions and technologies (local VirtualBox, to ProxmoxVE/OVZ and to AWS) and even hardware. As long as I ensure that I put things in the right places to start with I don't have to think about what other things I may have changed or tweaked (or database changes). It all just takes care of itself.

Don't get me wrong, I still use git as well for development but I use that more for smaller projects.

[edited to improve clarity]

Jeremy Davis's picture

I just edited my previous post to make my intention clearer.

And yes you may need to do a little tweaking to initially migrate your data from your old Redmine appliance. Personally I'd do that with a backup (as discussed above) but leave the original server unlocked and document all the steps. This means you'll need to do it twice, but gives you the room to take your time reducing stress and presure to do it quickly (the old site can keep chugging away merrily).

Then once you have it all documented and running sweet (on your dev VM) repeat the process but this time lock the original server while you do it. It should be quick and easy second time around (as long as your documentation is good!).

L. Arnold's picture

What you are trying to do should not be complicated at all.  I just went through something similar.

1:  Download a VM Package from TKL Website  (say Redmine)

2:  Install the VM Install as a new VM.  Configure it w/ IP addresses as needed.

2:  Modify it as you like (say with a new package)

3:  Run TKL-BAM backup to a backup image

4:  Install a second version to a separate New VM.

5:  Run TKL-BAM Restore to the new VM.   You will then have a second VM configured like the first VM but assumably with its own IP Address and SSL id.  Passwords, at this Point will be the same on both systems.

6:  Run Turnkey-Init (Not sure the actual program term just now) but this will re apply all your startup passwords and give a unique, rather than identical setup.

This is the standard way TKLBAM works rather than something unique.

Perhaps I am not understanding the special need you are trying to create that is not outlined here.  I have been recieving emails on this subject as I posted some time ago in this thread and my understanding of the problem is perhaps cursory.  Likely best for this to be in a new Thread rather than the original TKLBAM thread.

I hope this helps.

L. Arnold

 

L. Arnold's picture

If you don't want to run TKLBAM-Restore to build subsequent builds you could use TKLPatch to build an ISO which will build fresh VM's for you.

Assuming your source environment does not change (as it just did for OpenERP) you could then run the basic Patch structure on new versions of our packages that you are bundling with the Patch Process and move forward with fresh subsequent generations.

One of the difficulties is going from one Generation TKL to another as well.  Ubuntu to Debian may break your patch (without changes to the patch itself) but likely Future TKL generations will be Debian so if you start w/ ver 12 you should be able to move forward somewhat smoothly w/ TKLPATCH.

easies though is sticking with standard TKL Appliances and your Modifications in your ecosystem via TKLBAM.  If you are going to share your updates, TKL Patch is probably easiest, though you could have two accounts in the TKL world and backup (via TKLBAM) to the two accounts w/ TKLBam.  More than that would be brain damaging I expect.

Frank's picture

all was working fine untill my credit card expired and after fixing the nex credit card info I developed problems, on the dashboard everything seems allright, it even says that is doing backups and all but at my Mysql applaence

 

when I try to install my api key on the Link to my TurnKey Hub account I get this error

 

Error: (6, "Couldn't resolve host 'hub.turnkeylinux.org'")


Pages

Add new comment