SUGGESTION - FULL TKBLM backup option to cover systems incompatible with the current TKBLM backup

You know that TKLBAM is completely customisable right? You can include/exclude whatever you like. Also using the TKLBAM hooks, you can run stuff before/after backup/restore. Have a read of the docs.

What could be really cool is if you were to work out what needs to be included to make say Cherokee backup ok and once tested and confirmed we could include that info in the docs.

So you can cutomise TKLBAM to do exactly what you want, without creating a 'full' backup. In fact because of the way TKLBAM works, backing up certain parts (such as areas of the filesystem handled by package management) of your server will potentially create issues and break stuff.

However, if you are intent on creating a full backup, then TKLBAM may not be the ideal way to do it. Most TKL installs utilise a technology called LVM. It is possible to take a full filesystem snapshot using LVM, which you could then put anywhere you want (eg upload to S3). I don't know enough about it to give an example but you should be able to find plenty enough info via google.

It's about migration as well! (TurnKey Linux Backup And Migration). How would a 'complete' backup work when migrating from v11.x (Ubuntu Lucid based) to (the yet to be released) v12.x (Ubuntu Precise based)? It may still require some tweaking (to allow for differences in package versions) but should basically work. What about migrating from a physical machine or AWS instance to an OVZ template? TKLBAM will theoretically even allow migration from an Ubuntu based system to a Debian based one (although again will probably require some tweaking as some packages are named differently between the 2 OSs).

So don't get me wrong. I'm not saying that your suggested aim is not possible or reasonable (but perhaps I didn't make myself clear). Quite the opposite; your suggestion/request of "a FULL server backup to S3" is definately feasible and reasonable, but it is well outside the scope (and aims) of TKLBAM and would require a completely different approach and mechanism. A full image is doable (via an LVM snapshot), but it precludes (or at least reduces) the possibility of migrating between platforms (machine or OS) because it is an exact copy of the full HDD (virtual or otherwise) which will include components that are inherant to that specific system only and uses a medium which won't necessarily apply to other environments. As I posted above LVM snapshots of the whole filesytem can be created no worries but whether or not (and how) they can be restored to different environments I don't know (that would require knowledge and experience that I don't have).

To try to acheive your aims using TKLBAM would be problematic. As I hinted in my post above, copying some areas (such as those handled by package management) over the top of a new system is not a good idea. Besides being in contravention of the Debian FS policy (which is defined for good reason) it will produce unreliable results and eventually lead to a broken system. You could set it up today and you can test it and it will probably work, but next week (when some packages have been updated) you will possibly find that it no longer works. And eventually it will break but you have no way of knowing when that may be. TKLBAM is about producing reliable results. If you set it up today and confirm your backup works, it will work tomorrow, next week and even next year (not accounting for the possibility of TKLBAM bugs and/or 3rd party bugs and/or other factors outside of TKL control).

Besides, I don't really understand why "being 'completely customisable' doesn't cut it"!? Even though TKLBAM "cannot know the intricacies of my server and thus reproduces a dud", surely you know the intracacies of your server? I would assume that if you have customised it you know the customisations that you have done and can reproduce them!? Assuming that it the case, customising TKLBAM to work in your scenario should be completely doable. It is not reasonable IMO to expect TKLBAM to account for every individual's preference (just as IMO it is not reasonable or feasible for TKL appliances to be created to every users specifications OOTB). TKLBAM fits with the overall TKL philosphy. Neither the TKL appliances, nor TKLBAM itself are intended as a 'one size fits all' solution nor as an end point that you must adhere to. TKL (and by extension; TKLBAM) are intended as fair and reasonable starting points on which to build (as you obviously understand due to your customisations). They are designed to be useful to newbs OOTB but customisable to those of us who wish to.

Personally I have quite heavily customised systems and successfully backed them up (and restored them) with TKLBAM. Its just a case of ensuring that when you customise the system you do it in such a way that it is ln line with Debian FS policy and include any files/folders that are not included with the TKLBAM backup set by default. If you are documenting the customisations as you do them (as 'best practice' suggests you should), then customising TKLBAM to take them into account is a relatively straight forward process. You can check what is being included by default by running a TKLBAM test and checking the log and ensure that your custom files and locations are included (and perhaps some specific locations excluded).

For example if you have a package that is installed from a 3rd party repo (that isn't included with TKL by default). You just need to make sure that you use a TKLBAM pre-restore hook to write the custom sources.list entry to your server. As long as the custom sources.list is in place prior to restore then TKLBAM will automatically install the package (as you have discovered by default TKLBAM will only use the standard repos). Any custom settings that you have should be in /etc/<package-name> and you may need to ensure that they are included (although if IIRC TKLBAM includes all of /etc). Even an app that is installed from source will work fine as long as you install it to an appropriate place and include the path in your TKLBAM set.

So in summary: I don't think bending TKLBAM to do what you want is feasible (nor would it be reliable). But the suggestion of a backup mechanism to take a full snapshot is reasonable (and would be reliable) but would require a completely different approach that is well outside the scope (or aims) of TKLBAM.

But TKL being open source and all, if you want to develop your idea and push it forward as a possiblity, then perhaps you could do some experiementation with LVM snapshot creation and restoration and post your results?

There are inherent potential problems when you choose to re-create the system (as TKLBAM does) vs. having the actual system.

As an option/suggestion, it seems valid, because sometimes you can save space/time and re-create if something goes wrong, and other times you can't risk it and/or it doesn't work.

And with nice storage prices from S3, an option that still doesn't break the bank.

Glad to help clear it up for you.

This explains a lot more as to the focus of TKLBAM and it's nature so to speak - thank you.

I suppose it's a matter of selecting a different system for certain backups that needs to be a full snapshot and use TKLBAM for lots of other things.

I generally like the TKLBAM methodology and will try to fit my needs into it's framework. If I get the Cherokee webserver worked out, I'll post info here.

Thanks again.

Just in case you missed the announcement! See the blog post. As it says, it only applies to EBS backed instances but it does what you were hoping for.