TurnKey Linux Virtual Appliance Library

TKLBAM: a new kind of smart backup/restore system that just works

Drum roll please...

Today, I'm proud to officially unveil TKLBAM (AKA TurnKey Linux Backup and Migration): the easiest, most powerful system-level backup anyone has ever seen. Skeptical? I would be too. But if you read all the way through you'll see I'm not exaggerating and I have the screencast to prove it. Aha!

This was the missing piece of the puzzle that has been holding up the Ubuntu Lucid based release batch. You'll soon understand why and hopefully agree it was worth the wait.

We set out to design the ideal backup system

Imagine the ideal backup system. That's what we did.

Pain free

A fully automated backup and restore system with no pain. That you wouldn't need to configure. That just magically knows what to backup and, just as importantly, what NOT to backup, to create super efficient, encrypted backups of changes to files, databases, package management state, even users and groups.

Migrate anywhere

An automated backup/restore system so powerful it would double as a migration mechanism to move or copy fully working systems anywhere in minutes instead of hours or days of error prone, frustrating manual labor.

It would be so easy you would, shockingly enough, actually test your backups. No more excuses. As frequently as you know you should be, avoiding unpleasant surprises at the worst possible timing.

One turn-key tool, simple and generic enough that you could just as easily use it to migrate a system:

  • from Ubuntu Hardy to Ubuntu Lucid (get it now?)
  • from a local deployment, to a cloud server
  • from a cloud server to any VPS
  • from a virtual machine to bare metal
  • from Ubuntu to Debian
  • from 32-bit to 64-bit

System smart

Of course, you can't do that with a conventional backup. It's too dumb. You need a vertically integrated backup that has system level awareness. That knows, for example, which configuration files you changed and which you didn't touch since installation. That can leverage the package management system to get appropriate versions of system binaries from package repositories instead of wasting backup space.

This backup tool would be smart enough to protect you from all the small paper-cuts that conspire to make restoring an ad-hoc backup such a nightmare. It would transparently handle technical stuff you'd rather not think about like fixing ownership and permission issues in the restored filesystem after merging users and groups from the backed up system.

Ninja secure, dummy proof

It would be a tool you could trust to always encrypt your data. But it would still allow you to choose how much convenience you're willing to trade off for security.

If data stealing ninjas keep you up at night, you could enable strong cryptographic passphrase protection for your encryption key that includes special countermeasures against dictionary attacks. But since your backup's worst enemy is probably staring you in the mirror, it would need to allow you to create an escrow key to store in a safe place in case you ever forget your super-duper passphrase.

On the other hand, nobody wants excessive security measures forced down their throats when they don't need them and in that case, the ideal tool would be designed to optimize for convenience. Your data would still be encrypted, but the key management stuff would happen transparently.

Ultra data durability

By default, your AES encrypted backup volumes would be uploaded to inexpensive, ultra-durable cloud storage designed to provide %99.999999999 durability. To put 11 nines of reliability in perspective, if you stored 10,000 backup volumes you could expect to lose a single volume once every 10 million years.

For maximum network performance, you would be routed automatically to the cloud storage datacenter closest to you.

Open source goodness

Naturally, the ideal backup system would be open source. You don't have to care about free software ideology to appreciate the advantages. As far as I'm concerned any code running on my servers doing something as critical as encrypted backups should be available for peer review and modification. No proprietary secret sauce. No pacts with a cloudy devil that expects you to give away your freedom, nay worse, your data, in exchange for a little bit of vendor-lock-in-flavored convenience.

Tall order huh?

All of this and more is what we set out to accomplish with TKLBAM. But this is not our wild eyed vision for a future backup system. We took our ideal and we made it work. In fact, we've been experimenting with increasingly sophisticated prototypes for a few months now, privately eating our own dog food, working out the kinks. This stuff is complex so there may be a few rough spots left, but the foundation should be stable by now.

Seeing is believing: a simple usage example

We have two installations of TurnKey Drupal6:

  1. Alpha, a virtual machine on my local laptop. I've been using it to develop the TurnKey Linux web site.
  2. Beta, an EC2 instance I just launched from the TurnKey Hub.

In the new TurnKey Linux 11.0 appliances, TKLBAM comes pre-installed. With older versions you'll need to install it first:

apt-get update
apt-get install tklbam webmin-tklbam

You'll also need to link TKLBAM to your TurnKey Hub account by providing the API-KEY. You can do that via the new Webmin module, or on the command line:

tklbam-init QPINK3GD7HHT3A

I now log into Alpha's command line as root (e.g., via the console, SSH or web shell) and do the following:

tklbam-backup

It's that simple. Unless you want to change defaults, no arguments or additional configuration required.

When the backup is done a new backup record will show up in my Hub account:

To restore I log into Beta and do this:

tklbam-restore 1

That's it! To see it in action watch the video below or better yet log into your TurnKey Hub account and try it for yourself.

Quick screencast (2 minutes)

Best viewed full-screen. Having problems with playback? Try the YouTube version.

The screencast shows TKLBAM command line usage, but users who dislike the command line can now do everything from the comfort of their web browser, thanks to the new Webmin module.

Getting started

TKLBAM's front-end interface is provided by the TurnKey Hub, an Amazon-powered cloud backup and server deployment web service currently in private beta.

If you don't have a Hub account already, request an invitation. We'll do our best to grant them as fast as we can scale capacity on a first come, first served basis. Update: currently we're doing ok in terms of capacity so we're granting invitation requests within the hour.

To get started log into your Hub account and follow the basic usage instructions. For more detail, see the documentation.

Feel free to ask any questions in the comments below. But you'll probably want to check with the FAQ first to see if they've already been answered.

Upcoming features

  • PostgreSQL support: PostgreSQL support is in development but currently only MySQL is supported. That means TKLBAM doesn't yet work on the three PostgreSQL based TurnKey appliances (PostgreSQL, LAPP, and OpenBravo).
  • Built-in integration: TKLBAM will be included by default in all future versions of TurnKey appliances. In the future when you launch a cloud server from the Hub it will be ready for action immediately. No installation or initialization necessary.
  • Webmin integration: we realize not everyone is comfortable with the command line, so we're going to look into developing a custom webmin module for TKLBAM. Update: we've added the new TKLBAM webmin module to the 11.0 RC images based on Lucid. In older images, the webmin-tklbam package can also be installed via the package manager.

Special salute to the TurnKey community

First, many thanks to the brave souls who tested TKLBAM and provided feedback even before we officially announced it. Remember, with enough eyeballs all bugs are shallow, so if you come across anything else, don't rely on someone else to report it. Speak up!

Also, as usual during a development cycle we haven't been able to spend as much time on the community forums as we'd like. Many thanks to everyone who helped keep the community alive and kicking in our relative absence.

Remember, if the TurnKey community has helped you, try to pay it forward when you can by helping others.

Finally, I'd like to give extra special thanks to three key individuals that have gone above and beyond in their contributions to the community.

By alphabetical order:

  • Adrian Moya: for developing appliances that rival some of our best work.
  • Basil Kurian: for storming through appliance development at a rate I can barely keep up with.
  • JedMeister: for continuing to lead as our most helpful and tireless community member for nearly a year and a half now. This guy is a frigging one man support army.

Also special thanks to Bob Marley, the legend who's been inspiring us as of late to keep jamming till the sun was shining. :)

Final thoughts

TKLBAM is a major milestone for TurnKey. We're very excited to finally unveil it to the world. It's actually been a not-so-secret part of our vision from the start. A chance to show how TurnKey can innovate beyond just bundling off the shelf components.

With TKLBAM out of the way we can now focus on pushing out the next release batch of Lucid based appliances. Thanks to the amazing work done by our star TKLPatch developers, we'll be able to significantly expand our library so by the next release we'll be showcasing even more of the world's best open source software. Stir It Up!

You can get future posts delivered by email or good old-fashioned RSS.
TurnKey also has a presence on Google+, Twitter and Facebook.

Comments

Great job!

Great job!

Awesome, sounds like magic!

Sounds like the closest thing to magic I've ever heard. Can't wait to give it a try...

I've been waiting for this announcement, see my comment on the "backups are hard" post.

Basil Kurian's picture

Congratulations !

Congratulations !


Congratulations!!!!

This is incredible news for TKL Community! Thanks for the hard work on that, it's really a wonderful backup system. It gives a great amount of added value to TKL. 

Jomy Muttathil's picture

Congratulations

Very Impressive!

Is there be an option for automated incremental backups?

I like how it is not just for backups but can be used for migration as well.  Very clever.

 

This is Huge!

Liraz Siri's picture

Automating incremental backups

To enable daily incremental backups you just do this:

chmod +x /etc/cron.daily/tklbam-backup

The FAQ's usage section explains the difference between full and incremental backups and how to configure how frequently a full backup happens (in between full backups all backups are incremental).

Liraz Siri's picture

Thanks for the kind words guys

We've been working on this harder than we've worked on any other TurnKey feature. It feels wonderful to be able to finally share TKLBAM with the world and it means a lot to us when people get as excited about it as we are!

Only on EC2?

Wow - huge.

Just to clarify, can I:

a) just use EC2 as a 'backup hub', and run my production and dev machines myself;

b) (preferred) run everything myself - ie a private version of the hub. 

 

Liraz Siri's picture

Answers

a) yes. You can run production/dev machines anywhere you want and use the Hub just for backup management. The Hub can also launch servers in the Amazon EC2, but that's separate from the backup stuff. In the future we would like to support other public clouds, and eventually even private clouds.

b) We don't yet support running private Hub instances (e.g., behind your firewall in a private cloud). Maybe in the future...

is it possible to backup it

is it possible to backup it to UEC and not only using Amazon ec2 ?

Liraz Siri's picture

Yes you can use TKLBAM with UEC

Amazon S3 is the cloud storage service. Amazon EC2 is cloud server service.

The distinction is important because you can already use TKLBAM to backup instances running on UEC (or anywhere else). If you use the default "automatic" storage target your backups will be stored on Amazon S3. But you can specify an --address option to store your backups anywhere. See the FAQ for details.

PS: Sorry for the late reply. Your question slipped through the cracks somehow.

Sincere Congratulations.

If I understand the documentation correctly, TKLBAM may be an essential part of our data management within the next week. I'll be rereading a few times.

Again, I realize how important this software is to the community and have only the sincerest congratulations, thanks, and admiration.


Well done!

Congratulations and thanks to the development team!

TKLBAM looks great guys but....

I cannot put anything out in the cloud - only locally - will this happen anytime soon?

Regards

Richard

Liraz Siri's picture

You can use TKLBAM locally without the Hub or Amazon S3

You just won't get to enjoy the streamlined usability enabled by the Hub, so you'll have to manage backups, keys and authentication credentials by hand.

See the FAQ for details.

Anyhow, there is no schedule yet for streamlined usability in a private cloud type usage scenario. We'll revisit this after the Lucid release. We have limited resources, so we have to pick our battles carefully.

Jomy Muttathil's picture

The logical next step

The logical next step would be a Turnkey Linux TKLBAM Appliance.

A TKL appliance for backing up and migrating other TKL appliances.

Everything comes full circle.

Seconded !

Seconded !


Excellent!

This looks great and would really simplify our backup situation! We were just talking today about setting up some Turnkey Linux VMs for a couple of our websites and I think we'd love to use this service. Went ahead and submitted my request to join the beta. :-)

Data Costs

I'm wondering what we could expect to see as far as data transfer and request fees go when using TKLBAM. The $.15/GB storage fee is pretty easy to estimate, but I remember from a previous blog post that some S3 backup options could easily create huge data transfer and request charges with relatively small amounts of data (because of block size, etc.). How has this been addressed with TKLBAM? I apologize if it's already somewhere in the documentation and I missed it.

Liraz Siri's picture

negligible request charges + volume size is configurable

You probably mean this blog post. You're talking about request charges which are $0.1 per 1000 PUT requests or per 10,000 GET requests.

By default the volume size is 50MB but you can configure that to anything you want. That's one PUT request per 50MB so it's a reasonable default that will generate negligible request charges in typical usage (e.g., $1 per 500 GB after compression).

Sounds good to me

Ok, great! You all seem to have done a great job with this, and we're definitely looking forward to it.

Some folks are wary of storage in the cloud, but a reliable service like S3 looks great for us. We have very limited bandwidth between our main office and our branches, so doing off-site backups is really difficult. We do however have a nice fat pipe from our main office to the outside world.

It looks like this is only

It looks like this is only for Linux servers, not Windows servers.

I think there's way more Windows servers than Linux servers out there...

Thank you, Tom

Jomy Muttathil's picture

Doh!

You made me spit out my coffee!

Liraz Siri's picture

The TurnKey approach to backup wouldn't work on Windows

I see two problems porting TKLBAM (or equivalent) to Windows:

lack of package management: The main problem is that Windows doesn't have a package management system or strictly enforced standards about where you put things. That means you have to backup the entire system, and lugging around gigabytes, even on high performance networks isn't nearly as mobile as lugging around a few megabytes, or even kilobytes!

For a fresh installation of a TurnKey appliance the backup is literally a few kilobytes.

Culture of proprietary software: There's also licensing issues and the culture of proprietary software on Windows. To most people that probably sounds like a bunch of mumbo jumbo free software ideology, but it's actually a very practical issue. TKLBAM would have been very expensive to implement if we had to reinvent the wheel rather than stand on the shoulders of giants to reach just a little bit higher. We leveraged a ton of amazing, mature open source software "for free". If you can't do that you're pouring huge amounts of capital into developing everything yourself. Even if your team manages to pull it off (programming lots of complex stuff is hard) you still need to get a return on that investment somehow and giving everything away as open source turns into a career limiting move. So then your software has to be proprietary, which further perpetuates the problem of reinventing the wheel over and over again. Rinse repeat.

Open source sidesteps that neatly, allowing tiny teams like us to innovate in a space dominated by 800 pound gorillas. Alas, the magic doesn't work well on Windows.

Jeremy's picture

Something of a workaround - use TKLBAM to backup Win data

But before I get to that Tom, I'd like to say that IMO you are very much wrong! Perhaps you may be right when thinking of home servers and/or small business servers (but even then I'm not sure). When it come to Enterprise and Web servers I think you'll find that Linux servers are extremely popular. It is quite hard to quantify as most Win servers are purchased whereas many Linux ones are not (or may even have been Win servers initially but then had Linux installed and so get counted as Win servers when they are in fact Linux ones). According to Steve Balmer back in Sept 2008 ""Forty percent of servers run Windows, 60 percent run Linux..." It's probably hard to say how that may have changed although many comentators have suggested that the GFC has increased the uptake of Linux Servers (due to lower TCO). Wikipedia has some interesting stuff to say about Server OS market share which is worth a read. Even if you look at market share figures alone, Linux servers are still a sizable minority and continue to have annual double digit percentage growth.

Sorry for the rant. Back to your original point regarding Win backups to the cloud...

Despite Liraz's comments about Win not being able to play nicely with the TKLBAM system, there is now (almost - have to wait until release) a potential workaround so you can use TKLBAM to backup Windows OSs. Using the TKL Bacula Backup appliance (currently only a beta patch for TKL Lucid Core beta - which does not yet support TKLBAM AFAIK) you can (will - hopefully!) have Windows client machines backup to the TKL Bacula appliance which in turn can then use TKLBAM to backup your Win data to the cloud. Sweet eh!!

Jeremy's picture

Typical!

I go away for a couple of days and don't check my emails and THIS happens!

But seriously - great work guys! TKLBAM looks impresive and builds nicely on the Hub. I think the name is fantastic too. TKLBAM has such a nice ring to it!

Also thanks for the kudos guys. I do it all for the love! I really enjoy feeling like part of the team and I feel honoured that I can be a part of such a great project! The community really seems to be growing and maturing of late which is awesome to see. Like you, I have been amazed with the TKLPatches that Basil & Adrian have been putting together.

Once again a worthy addition to the TKL brand!

Liraz Siri's picture

Thanks - we missed you!

You disappeared for a few days, so naturally Alon and I decided we had to do something immediately to lure you back. My idea was to fire up the Jed-Signal. Then Alon said, "I have a better idea, let's release that backup and migration system we've been working on for the past year". "You'll think it'll work?" I asked. "Let's just pray it does. Now quickly, to the TurnKey-mobile!"

I'm just glad it worked...

Liraz Siri's picture

TKLBAM works with TurnKey everywhere (including private cloud)

1) In automatic mode, you can use TKLBAM anywhere (e.g., private cloud) so long as you have network access to the Internet and can reach Amazon S3 and the TurnKey Hub. If your private cloud isn't firewalled off from the Internet it will work. In manual mode, you don't even need access to Amazon S3. You can store your backups on the local network. It's just not as easy to use.

2) TKLBAM depends on a few things that are currently TurnKey specific (e.g., a fixed starting point after installation / AKA the "appliance" concept). This isn't just some arbitrary limitation, but part of what powers the zero-configuration magic. Behind the scenes we've hand-crafted the default backup configurations for each appliance in the TurnKey library. In other words, we can make a very good guess at where you data is stored if you're using TurnKey File Server (for example) because we basically set it up for you.

Generic Ubuntu/Debian takes a different approach where the installation process is more general purpose. The installer asks you what kind of system you want, what additional packages to install, etc. So... no fixed starting point at installation.

But remember that under the hood TurnKey is just a series of Ubuntu (and soon Debian also) based systems optimized for a particular usage scenario. When you use TurnKey you are not making the choice not to use Ubuntu. You are making a choice to use an Ubuntu system we pre-integrated for you.

Linux Magazine article

Hi, you can find an interesting article about Turnkey Linux on Linux Magazine:

http://www.linux-mag.com/id/7857?hq_e=el&hq_m=1071524&hq_l=3&hq_v=fb6a4a01db

Good!

Alon Swartz's picture

Press coverage

Thanks for the link jgab, you actually beat me to the punch.

I'd like thank all the online publications and writers who have covered TKLBAM's announcement, and helping us get the word out. For your reading pleasure, here is the list of articles I came across, listed in order of publication date.

 
Let me know if I missed any.

can this be done locally and if so how?

For example, I have everything on one partitioned drive and I want to back it up to another partititoned drive on the same computer-can I do this without amazon? If so, how?

Jeremy's picture

TKLBAM is aimed at server Backup And Migration

Rather than individual folder backups so it may not really be what you're after. Perhaps consider some other method such as rsync (not sure if its installed in TKL by default but easy to install if not: apt-get update && apt-get install rsync). Iif you want a regular backup, consider writing a basic bash script using rsync and set it as a cron job.

Another option that may suit better is the proposed TKL Bacula appliance. Its only a TKLPatch at the moment but patches are relatively easy to apply (and it would help us do some testing if you give it a go). Hopefully it will get released as an offical TKL appliance next release. Its a lengthy thread but worth reading if you're keen for a sitewide backup solution.

Back to TKLBAM; if you'd like to persevere with a local TKLBAM backup, have a look at this post I did earlier.

Liraz Siri's picture

Any Duplicity storage target is supported

In manual mode you can use any storage target supported by Duplicity, and that includes the local filesystem and many others (e.g., rsync, SFTP, rackspace cloud files, webdav, IMAP). The local filesystem is actually one of the easier storage targets to use because you don't need to mess around with authentication credentials.

So assuming you want to store your backup at /mnt/otherdisk:

tklbam-backup --address file:///mnt/otherdisk/tklbam/backup
tklbam-escrow /mnt/otherdisk/tklbam/key
And restore like this:
tklbam-restore --address file:///mnt/otherdisk/tklbam/backup \
               --keyfile=/mnt/otherdisk/tklbam/key

Not as easy as the Hub-enabled "automatic" mode, but still vastly easier than your conventional backup process. The disadvantage is that you won't be able to restore/test your backup in the cloud, or from a VM running in another office branch. Also keep in mind that a physical hard disk, even a RAID array, provides much much lower data reliability compared with Amazon S3.

For this reason we recommend users use local backups to supplement cloud backups (e.g., providing fast local access).

TKLBAM vs TKLPATCH for Appliance development

Hello,

TKLPATCH development (as I've understood it)

If I understand things well, TKLPATCH essentially are BASH/SH code that help transform a vanilla Ubuntu/Debian VM into a specific appliance. The way to developping a TKLpatch would basically be : install stuff by hand, then recode than in BASH so that everything happens automagically when the TKLPatch is run.

The end result would be twofolds :
- you can download a ready to use appliance, built with the TKLPATCH for you
- you can look at the related TKLPATCH BA/SH code and modify it

TKLBAM development (guesses)

TKLBAM can be used for development as shown in the TKLBAM FAQ. Modification of an image is by hand by apparently no need to note down/script all the steps in TKLPATCH BA/SH files.

The end result is twofolds but slightly different as above :
- you could download a ready to use appliance, built out of some backup+vanilla VM for you,
- you could look at the appliance "definition code" by reading some TKLBAM backup summary file... sort of like a TKLPATCH BA/SH code except that it's harder to modify it by hand...

Short conclusion

In the end, in terms of appliance development, it seems on the one hand that TKLBAM lets you develop reusable appliances quicker than TKLPATCHES appliances because you don't have to write scripts. On the other hand, TKLBAM appliances construction do not seem to be that much easily forkable/reviewable for development, compared to TKLPATCHES.

That was aloud thinking and I did not dig deep because I do not know both topics well. I'm sure some things are wrong in my reasoning. How do you TKL admin people see a best way for reusable/shareable/customizable TKL appliances development ? Would that definitely remain TKLPATCH or TKLBAM or something else ?

Liraz Siri's picture

TKLBAM for assisting appliance development - on the radar

You certainly could use TKLBAM to save the state of a customized appliance and then restore it elsewhere. You wouldn't have to learn how to use TKLPatch to do that.

On the flip side, TKLBAM backups are currently not shareable, and the result of a backup is not as transparent. A well structured TKLPatch is easier to understand the log of a restore operation.

You're right however that there does seem to be some overlap and in fact we are thinking about how to leverage TKLBAM for appliance development in the future.

I discussed one idea (automatic TKLBAM to TKLPatch conversion) in another thread:

Alon and I hope to accelerate that by making substantial improvements to the community development infrastructure. Imagine for example doing an upgrade manually (e.g., in the cloud or in a VM) and then being able to export the result automatically to a rough TKLPatch. We can leverage TKLBAM to do most of the heavy lifting (e.g., identifying the delta).

Bottom line, we want TurnKey to work smarter, not just harder. Instead of solving problems by brute force we can come up with creative, innovative solutions that gradually eliminate the friction that is currently preventing most of our users from contributing back to the project. This should allow the project to develop in a more decentralized fashion that empowers and leverages the community to do much more. Our goal is to support a community with hundreds of active contributors.

tklbam / tklpatch easy developement all in all it's like macros

Hello,

thank you very much for your reply !

I'm happy to see that my thoughts were not on wrong tracks.

Read your saying "Imagine for example doing an upgrade manually (e.g., in the cloud or in a VM) and then being able to export the result automatically to a rough TKLPatch. ", made me thought in the end that a name to this ability is just being able to create linux system actions macro code : you do things on the system and you see the equivalent .bash/.sh code for what you did in most real time. Then you can customize that code by hand.

Current TKLBAM diffing probably cannot tell what action has been achieved at what moment or in which order, and exactly what command line tool & parameters provoked which changes.
For TKLPATCH auto-generation (macro recording), a first implementation idea would be to watch both bash_history and config file edits, and dump the noticed files in a separate future TKLPATCH script file.

I am not a TKL core developer and have only started trying to build an OpenStreetMap appliance aside. I apologize if I can just comment aloud here but not contribute (yet) to help implement in some direction what I'm reflecting here.

Take care !

Long View

Hello,

I've been trying out tklbam for a couple of weeks and I love it; thank you for building such an elegant and robust backup solution!  
 
I have a question about restoring data from the hub after, for example, many years have passed and the appliance versions have changed via significant yearly upgrades. How would this work? (I didn't find info on this in the FAQ, apologies if I missed it.)
 
In other words, how would I be able to restore a back-up that was created from an Ubuntu Hardy appliance to future appliance version?  I did see that we will be able to restore Ubuntu Hardy to Ubuntu Lucid, which is great. But, how about restoring Ubuntu Hardy to a new TKL appliance five or more years into the future?  How many versions back will new appliance versions be compatible with? I know it would be ideal to always upgrade our appliances with each new major release, but I also know the reality of our organization: doing "the ideal" tends to be rare.  
 
Thank you,
TG
Liraz Siri's picture

TKLBAM doesn't care what version you are restoring to

Thanks for the positive feedback! Getting good feedback from our users means more to us than you might imagine.

Regarding your question, TKLBAM only cares what version of a TurnKey appliance you're using when you backup, because it uses the profile for that version to calculate the delta from the installation state. In other words, it figures out what files have changed, what packages have been installed, etc.

But TKLBAM doesn't (currently) care what version you are restoring to. It just applies the delta in a generic way.

For example, if you changed a configuration file in /etc/php5 and restore the backup to a system in 2015 using PHP6 the file you changed will just be restored to /etc/php5. That's the way it should be as your PHP5 configuration file might not even make sense in the context of PHP6.

On the other hand, if you're still using PHP5 by then, just a newer version, most likely your old configuration file will still work in that location.

Upgrading to major new versions of applications that break backwards compatibility is probably still going to require some manual effort. It's not part of the current design but I wouldn't be surprised if in the future we added logic to detect and handle common upgrade scenarios more easily.

Restoring just your data to a new system in 10 years time shouldn't be an issue. Worst case scenario you can just exclude certain "problematic" directories such as /etc when you restore like this:

# this restores everything except changes to /etc
tklbam-restore 1 --limits="-/etc"
BTW, taking the long view is another reason I never felt comfortable entrusting my backups to proprietary solutions. What happens if the vendor disappears? With an open source solution, users can band together to help themselves if that happens. If a copy of your backup volumes and encryption keys are still around in 100 years time, your grandchildren will be able to hire an open source software archaeologist to extract your data from it.

But that's probably a longer view than you are interested it...

Jeremy's picture

Hehe :)

If a copy of your backup volumes and encryption keys are still around in 100 years time, your grandchildren will be able to hire an open source software archaeologist to extract your data from it.

But that's probably a longer view than you are interested it...

Awesome! :)

Indeed that's the power of

Indeed that's the power of Open Source... but will the cloud storage provider you rely on stil be there ? Will your user friendly, featureful Hub web UI still be available ? That's another story !!!

This leads back to Jomy's post above... next step should be a TKLBAM appliance.

Anyway, thanks a bunch, keep on the good work.

Bests


L. Arnold's picture

The tklbam-restore syntax above needs to be expanded

I just spent about an hour working out how to use the "--limits" command properly.  As I can't easily "copy / paste" out of MY SSH panel (which is running a big Restore Now anyway)..  I thought I would throw this out there...  Seems Simple here but I kept following the instructions in the Docs adding things like semicolons, and Limit# declarations, and multiple parenthesis sets...  This method below started a restore excluding a few directories and one suspect SQL file:

tklbam-restore 12 --time 2011-10-20T06:33:38 --limits="-/var/www/appname/var/cache -/var/www/appname/var/sessions -mysql:appname/messedupsqlfile" --force

Not sure that the Force is needed there, but in this situation I was clearing the chafe from DNOS(sic) attack (or series of them.  Also, it seems important that we start talking about how to really block the aps from sql injections techniques...  (mine seems to have come from repetitive and "big block" queries going off of our search blocks...

Thanks for building a system that can deal with this!

Good Job!

You have make the backup solution more easy then ever!

Can TKLBAM backup my Windows hard drive?

In my imagination, there is a TKLBAM live CD that copies all used blocks of a partition to the cloud, and restores them.

Jeremy's picture

TKLBAM only works with TKL appliances OOTB

But I think it's a really interesting usage case and something that may be worth further consideration by the Devs?

In the meantime I would imagine that you could boot a TKL CD as a live/demo instance (or as a dual boot setup or physically connect your Win HDD to a TKL appliance) and you should be able to mount your Win partition/HDD in TKL and then back it up using TKLBAM. It will involve a little commandline trickery and possibly some trial-and-error (eg where to mount it so it gets backed up) but should be possible and not too difficult. If you go this path and get it to work it'd be great if you can document the steps you take as no doubt others would be interested.

The other option would be look at the TKLPatch for Bacula (hopefully it will be released as an official TKL appliance soon) and backup Windows to that and then use TKLBAM to backup the data. AFAIK Bacula will allow you to do a live backup (ie with Win running) sotheoretically you could do that with TKL Bacula running in a VM. Not sure how practical that would actually be?

So can I use TKLBAM to import my existing VPS deployments?

Hi everyone.  I just learned about the work you guys are doing and am in awe.  I am also both excited and relieved to have found that you have developped exactly the product I'm looking for - I think.  My question is simple:  How do I get my existing deployments into TKL?  I've spent a few hours reading through stuff and haven't seen a definitive answer (some getting started docs are needed, and in the spirit of Open Source I'm willing to pitch-in if help is needed).

It seems like I would use TKLBAM to import the file structures lock stock and barrel. Is that correct?

I have 2 production instances running at Linode, both on Ubuntu 10.04.  One is a WordpressMU install, the other is a custom app built on django. 

So would I start with a couple of ubuntu instances on the hub and go from there?

And sorry if this question is already clearly answered somewhere, but I've looked thoroughly and not found the answer.  If the answer is lying around soemwhere, I'd humbly suggest putting it soemwhere front and center where it's easy to find.

Thanks!


Jeremy's picture

Unfortunately not - TKLBAM is TKL specific

TKLBAM can only be used for migration between TKL appliances (eg from the Hardy based v2009.10-2 to the new v11.0). That means that you cannot do what you are hoping to.

I would suggest that you install TKL (to wherever) and rsync from your existing server to the relevant TKL instance (WordPress and Django I would imagine). Hopefully you'll be lucky and it all works smoothly. Once you have it all transferred accross and working sweetly you can then use TKLBAM to back it up.

As for docs, yes you are right. We need more and better! Adrian has made a start on tidying that up here on the Dev Wiki. Please feel free to put some rough drafts, links to forum threads you find useful and/or any info you think should be in the official docs (you will need to register/login seperately for the Dev Wiki).

Wow, thanks for the quick reply

OK, so I should just set-up a local instance of TKL Core and rsync with fingers crossed? 

One concern is that there have been some server tweaks done on both installs but especially the django app so I'd like to just pick up the whole system and plunk it down if I could.

I guess I've just recommended another product for you guys...

I know VMWare has a client that does that, it's called the configurator or something like that.


Jeremy's picture

I would use the relevant TKL appliance for each

I would use TKL Django and WordPress appliances respectively (rather than Core). That way you will be able to take full advantage if TKLBAM for backups (TKLBAM will only backup what is needed - if you use Core you will have larger backups unneccesarily).

As TKL v11.x is based on Ubuntu 10.04, assuming that your current appliances are using default locations for content and settings, you should only have to rsync content and settings files and dump/import the MySQL database(s). If your current deployments are set up 'properly' then most, if not all of your tweaks should also transfer across no worries.

I am happy to try to assist you (but no promises about final outcomes) with more fine grained help but it's probably better to start a new forum thread (I'll see it, but feel free to post a link to your new thread here too).

Custom Core/LAMP/LAPP Appliances on Hub?

I think I overlooked an important detail about the hub.  Does the hub allow a custom appliance, lets say built on a Core/LAMP/LAPP appliance, to be properly backed up to the hub regardless of the type of application running on top?

Jeremy's picture

Yes, but...

The Hub will assume that it is the base appliance that you used and will backup user editable locations and will create a reinstall list of installed packages. Assuming you used the standard repos included with TKL and haven't put anything in 'naughty places' then it will all go according to plan.

However if you have additional repos or manually installed .debs then I won't be able to install them. Also you may wish to do manual overide to include specific paths that are not included by default. See Liraz's post here for more detail.

Best way to go is to test it out and have a bit of a play.

Thanks JedMeister

I started a new topic here: http://bit.ly/gMnti3

I proposed one method I thunk-up, and posted some other possible resources I found. 

I'd love to hear your thoughts, and am grateful for your offer to help. I'll take you up on it, and I totally absolve you of any responsibility for results.

Thanks again.


Any plans for incremental restore?

First off, contratulations on a great feature. I've been following your work here for some time but I've only just got around to trying it out; it's an impressive feature that you've created, well done!

One slight niggle though, the incremental backups are great, but the restore feature seems to download a full restore each time. I was thinking of using tklbm to keep 2 machines in sync (plus a remote backup), so a full download each time is a bit excessive. It occurs to me that if you kept a local copy of the last restore then you could use rsync to do an incremental restore (assuming the hub is capable of doing this of course). What do you think?

Thanks for all the great work.

Des


Liraz Siri's picture

Interesting idea, needs more thought

Thanks for the feedback Des. I'll have to think about this a bit more but this could be a good idea, even though TKLBAM was never designed for this specific usage scenario. It's impractical to satisfy all usage scenarios with a single, well designed tool, but if with a modest change we could provide system-level syncing capabilities that would be worthwhile.

Note however the implementation would be a bit more involved than you seem to suspect. TKLBAM stores backups on S3, which doesn't support the rsync protocol natively. Also, we use Duplicity on the back-end and Duplicity doesn't have support for this type of caching, though it may be possible to get around that without diverging more from the main Duplicity branch.

Mysql Does Not Work On Restore

I've tried to restore my LAMP appliance to EC2 as well as another blank Turnkey LAMP appliance using tklbam-restore, both attempts my Mysql does not work. Any ideas? Please help.

Removing Older Backups

It's great that you've built and documented how to add the backups. I don't yet understand how to remove them? I'd like to keep a nice rotation, but want to make sure I'm not backing things up on Amazon forever if I don't need to.

Jeremy's picture

How many full backups get kept is set in the Hub

If you go into the backups section and open the relevant server it can be set there (default is unlimited).

Say 1G Backup

What happens when you set a backup limit? What happens when you hit or cross that line?  Are old backups deleted automaticly?

Jeremy's picture

It's number of backups, not size

I've got mine set up to keep 2 backups. That means that once a third full backup is made the oldest one is automatically deleted. Note that this setting is per server (so they can be different).

Backup Rotation Policies

I like having backup rotations where if a problem isn't noticed for a few days it's possible to go back and restore stuff.  Ideally it would be possible to go back even a month.

I'd like to see:

  • daily incremental backups
  • full weekly backup
  • full monthly backup

I'm used to backing up with rsync, just focusing on /etc the drupal directory & mysql dumps

Jeremy's picture

How about a compromise?

I know it's not exactly what you are after but it's close.

Enable daily incremental backups. Set full backups for 2 weekly, and keep 2 full backups.

Then you get relatively short incremental chains and minimal storage usage, plus up to a month old full backup.

Liraz Siri's picture

Oldest backup chain is deleted

Yes, if you set a limit the oldest backup chain is deleted by the Hub automatically every time you create a new chain that exceeds the limit.

There are a bunch of common

There are a bunch of common cache files in Drupal that i'd rather not back up.  Does:

echo -mysql:*/sessions >> /etc/tklbam/overrides

or

echo -mysql:*/cache >> /etc/tklbam/overrides
echo -mysql:*/cache_* >> /etc/tklbam/overrides

work to override generic database tables?

Escrow Key

Hi there,

I'm just starting to try and set up TKLBAM and am stuck at the escrow key generation. The man page says:

tklbam-escrow [ -options ] KEYFILE

I'm not sure where the key file should go. My appliance is on a VM but there is only one disk. If I store the keyfile in /some/random/directory then wouldn't the key file get backed up with the rest of the data? In this case, is this the usual way to do it. I see also that the help for KEYFILE says:

KEYFILE: File path to save the escrow key (- for stdout)

This leads me to believe I can just print the key to stdout and copy it to a secure location elsewhere. However, when I issue the command:

tklbam-escrow -

...it just prints the help text. Am I going about this correctly?

 

Thanks for your help,

Jack

Jeremy's picture

I have no experience with using that manually

I always just use AWS (cause it's so easy and cheap - also it's good to have an offsite backup IMO). But I suspect your suspicion is correct. But to be on the safe side, rather than copy paste the key, I'd be inclined to copy the keyfile out using SFTP (so the full formatting of the file is preserved).

Hi Jeremy, Thanks for your

Hi Jeremy,

Thanks for your reply but I'm not sure you understood me correctly. I am trying to configure it with Amazon S3 (is that what you meant by AWS?) but any of the examples of generating the escrow key use e.g. /mnt/path/to/dir/on/second/disk

As I don't have a second disk I'm wondering where would be best to store it?

Thanks

Jeremy's picture

Ok sorry I think I get you now...

I don't bother with the extra layer of encrytion. I thought you were trying to run a backup to a local location (rather than S3/AWS - Amazon Web Services). Bottom line is - sorry I have no idea. I'd be inclined to just test it and see what happens. If you try restoring to a fresh AWS appliance it will be really quick (and cheap too) then you'll know.

Thank you Jeremy, I guess my

Thank you Jeremy,

I guess my main worry was storing the escrow key on the same volume that's being backed up. This would mean that the key to decrypt it is in the backup. However, that's probably ok since the backup would be encrypted anyway (if I understand correctly). Another thing I could do is exclude the directory with the key from the backup but since I haven't seen anything about that in the documentation then it's probably notrequired. Any googling just brings me to questions asking what the key if for also.

Could you recommend a place to ask exactly what the escrow key is for?

Thanks again,

Jack

Does restore work both ways?

I was also wondering if the restore works both ways?

Example:

In the Webcast and example usage scenario, 'Alon' configures a website on his local appliance (dev vm). Once he is happy with the configuration he backs it up to the cloud with TKLBAM and then deploys the website to his cloud VM (production vm). 

What isn't clear is whether he can then make a configuration change on the production vm and use TKLBAM to deploy them back to the original dev vm.

The reason I ask is that I already have an appliance on a VPS so would like to install a local VM and use TKLBAM create a local dev VM from which I can deploy in the future.

 

Thanks,

Jack

Jeremy's picture

Theoretically it should work

But in my experience TKLBAM works best on a clean instance. Perhaps try it and see how it goes. If it works, great. Otherwise start up a new clean TKL VM and restore to that.

I will be setting up a clean

I will be setting up a clean local TKL VM (dev vm) but as I already have some applications and configurations on my TLK VPS (production vm) I thought that I could use TLKBAM to grab the changes (since the initial clean TLK appliance install) and bring them to my local dev vm.

However I then want to use the local dev VM to make all changes in the future, and then deploy these to the production VM using TKLBAM. Im wondering if the backup command would get confused if it is called from two different install. Or is there a different command I should be using?

Thanks,

Jack

  The more I think about it,

 

The more I think about it, the more I get confused. It would make sense though that it should work the way I am proposing. Otherwise, after deploying his initial site to the live TLK appliance, 'Alon' would eventually want to make some changes, e.g. installing another app. If he were to do this on the local 'dev vm' and deploy to the live VM then his database would be overwritten. He would first have to backup the live VM and restore to the dev VM before making the configuration changes, after which he would then deploy back to the live VM. So it would function in a similar fashion to e.g Git.

However, I could be totally misinterpreting the use cases of TKLBAM...

Jeremy's picture

The way I'd do it...

This assumes that your starting point is a cloud server.

  1. Back it up with TKLBAM.
  2. Temporarily lock your cloud server (if it's a static site that's easy, if a more dynamic site - lock forums/comments etc).
  3. Deploy a fresh local vm and restore your backup to it.
  4. Make the changes you wish to make on your local VM, test etc.
  5. Make a new TKLBAM backup of it.
  6. Restore your new backup to your original server (or even create a fresh instance) and unlock it again (you'll need to reconfigure your DNS etc if you start a fresh instance).

Next time you want to make adjustments or test a new feature - whatever, just repeat the steps...

You could use git or something instead but I like TKLBAM because I can easily move customised servers between different VM solutions and technologies (local VirtualBox, to ProxmoxVE/OVZ and to AWS) and even hardware. As long as I ensure that I put things in the right places to start with I don't have to think about what other things I may have changed or tweaked (or database changes). It all just takes care of itself.

Don't get me wrong, I still use git as well for development but I use that more for smaller projects.

[edited to improve clarity]

That makes more sense, Could

That makes more sense,

Could I just clarify a little further if you don't mind....

In the scenario you have described, the initial backup of the cloud server is only used to restore the configuration to the local VM. When I then make the new changes to the local VM I create a new backup (which creates a separate backup entity on the hub) and restore that backup to the cloud VM. The original backup is effectively done with (assuming I can restore from the new one) and can be deleted.

I would then repeat this procedure every time I want to stage changes from the local VM back to the cloud. Is that correct?

What I really want to do initially is upgrade an old TKL Redmine appliance to an newer version of the TKL redmine appliance. I guess I'll have to deal with some issued though since redmine is installed from the upstream tarball...

Thanks again,

Jack

Jeremy's picture

Yep you got it! :)

I just edited my previous post to make my intention clearer.

And yes you may need to do a little tweaking to initially migrate your data from your old Redmine appliance. Personally I'd do that with a backup (as discussed above) but leave the original server unlocked and document all the steps. This means you'll need to do it twice, but gives you the room to take your time reducing stress and presure to do it quickly (the old site can keep chugging away merrily).

Then once you have it all documented and running sweet (on your dev VM) repeat the process but this time lock the original server while you do it. It should be quick and easy second time around (as long as your documentation is good!).

All clear now :)

Thanks very much for your help. That's cleared up a lot for me. Very much appreciated.

Also, my apologies. I'm new here and didn't intent add a load of comments. I should have used the forum. I will do in future.

Cheers,

Jack 

L. Arnold's picture

Chiming in here.

What you are trying to do should not be complicated at all.  I just went through something similar.

1:  Download a VM Package from TKL Website  (say Redmine)

2:  Install the VM Install as a new VM.  Configure it w/ IP addresses as needed.

2:  Modify it as you like (say with a new package)

3:  Run TKL-BAM backup to a backup image

4:  Install a second version to a separate New VM.

5:  Run TKL-BAM Restore to the new VM.   You will then have a second VM configured like the first VM but assumably with its own IP Address and SSL id.  Passwords, at this Point will be the same on both systems.

6:  Run Turnkey-Init (Not sure the actual program term just now) but this will re apply all your startup passwords and give a unique, rather than identical setup.

This is the standard way TKLBAM works rather than something unique.

Perhaps I am not understanding the special need you are trying to create that is not outlined here.  I have been recieving emails on this subject as I posted some time ago in this thread and my understanding of the problem is perhaps cursory.  Likely best for this to be in a new Thread rather than the original TKLBAM thread.

I hope this helps.

L. Arnold

 

L. Arnold's picture

You could also use Patch if you want to creat an ISO

If you don't want to run TKLBAM-Restore to build subsequent builds you could use TKLPatch to build an ISO which will build fresh VM's for you.

Assuming your source environment does not change (as it just did for OpenERP) you could then run the basic Patch structure on new versions of our packages that you are bundling with the Patch Process and move forward with fresh subsequent generations.

One of the difficulties is going from one Generation TKL to another as well.  Ubuntu to Debian may break your patch (without changes to the patch itself) but likely Future TKL generations will be Debian so if you start w/ ver 12 you should be able to move forward somewhat smoothly w/ TKLPATCH.

easies though is sticking with standard TKL Appliances and your Modifications in your ecosystem via TKLBAM.  If you are going to share your updates, TKL Patch is probably easiest, though you could have two accounts in the TKL world and backup (via TKLBAM) to the two accounts w/ TKLBam.  More than that would be brain damaging I expect.

Thanks

Thanks Arnold for your replies also. I don't think I'm ready for patching just yet. I just want to do the backup, upgrade and restore option.

Cheers,

Jack

problem after trying to reconect

all was working fine untill my credit card expired and after fixing the nex credit card info I developed problems, on the dashboard everything seems allright, it even says that is doing backups and all but at my Mysql applaence

 

when I try to install my api key on the Link to my TurnKey Hub account I get this error

 

Error: (6, "Couldn't resolve host 'hub.turnkeylinux.org'")


Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)