Downloadable TKLBAM Profiles for non-cloud based backups?

Hi AJ,

I haven't found a solution (valid set of readily available TKLBAM profiles without setting up an AWS account, etc. for online TKLBAM) yet, I was going to give this post a few weeks to see if it ferreted out an answer. 

Barring that I was just going to trying to handjam a profile until I got something working which, I will post here if/when I can get it working - the links you posted look like they'll be really helpful to that end, thanks!

Hmm, it looks like you are correct and it only contains what is explicitly included in the nextcloud profile?! That's a little embarrassing. You'd think that after 10+ years with TurnKey, I'd be all over TKLBAM...

In my defence, other than some fairly minor maintenance, I haven't worked much on TKLBAM and aren't particularly familiar with it's code base. Also, I don't (and haven't) actually used TKLBAM with a custom/self generated profile (I use it to back up my laptop, but I just have the core profile as a base, with custom overrides - servers I manage just use the default profiles via the Hub). Internally, when we build a new release, we have a custom script which generates the backup profile from the freshly built ISO (and does use the core profile as a base.

So I'll have to look into why 'tklbam internal create-profile' doesn't work as it should (and as the docs say it does). But in the meantime, in the case of Nextcloud, as the only path in the specific nextcloud profile is an old path that is no longer used, you could work around the issue by just using the core profile directly. Note that core includes /var/www (which will include both /var/www/nextcloud and /var/www/nextcloud-data). So whilst it's not really ideal, in the specific case of Nextcloud (and any other app that has an (effectively) empty profile), you could just use core's directly, e.g.:

tklbam internal create-profile ~/nextcloud_profile /turnkey/tklbam-profiles/core

Until I work out what is going on with TKLBAM's custom profile generation, if/when you do find yourself needing to build a profile for a server that does have entries in it's own profile, a workaround would be to combine the profiles first. E.g. (using fictional 'complex_server' appliance profile), something like this:

cat /turnkey/tklbam-profiles/core /turnkey/tklbam-profiles/complex_server > complex_server_profile_conf
tklbam internal create-profile ~/complex_server_profile_dir complex_server_profile_conf

As to your question about pre-populated dirindex file, ideally the custom backup profile should be generated from a fresh server (as we do with the aforementioned script that generates the profile from the freshly built ISO). That will allow you to restore a backup to a vanilla instance of the same server (i.e. migrate data). Although AFAIK it should work (for the purposes of backup restore to the same server) regardless. The profile generation should build a pre-populated dirindex file. I'm almost certain that the reason why it doesn't in your testing, is because the only path included in the backup profile (/usr/share/nextcloud) doesn't exist, so the pre-populated dirindex is empty! I hope that makes sense.

Finally, please be sure to test your backups and make sure that it's working as you intend, well before you need to rely on it. Remember that just like untested programs; untested backup config should be considered unstable and likely buggy! Hopefully it won't be, but until it's tested that should be the assumption. We highly recommend that you have a regular backup testing regime. I personally do quarterly "quick and dirty" backup tests and annual migration tests (i.e. restore backup to a fresh instance).

Thanks too for your kind words.

Hi Jeremy,

Than you very much for taking the time to thoroughly reply. I think this is a lot more clear now. I did actually complete option 1 prior to your response, but I would still like to use my own generated profile (mostly out of curiosity, and helping better understand tklbam).

Following your guide I was able to generate my own profile for nextcloud:

tklbam internal create-profile ~/nextcloud_profile /turnkey/tklbam-profiles/nextcloud

But I found that the dirindex.conf only contained

/usr/share/nextcloud

It did not include all of the other directories from core (and nextcloud) as I was expecting from your comment "They all inherit from core". If I create a profile for core, then I have all of these, but not /usr/share/nextcloud. I expected the profile to essentially merge these two because that is what I observed from the profile downloaded from the hub account use.

It is obviously trivial to edit dirindex.conf manually to include the nextcloud, but I wanted to make sure I wasn't missing an obvious step to do that automatically.

Lastly, I wanted to ask if you could clarify a misunderstanding I had about tklbam. I now understand that when creating a profile, rather than using a profile from the hub, you are missing out on some efficiency. Because the hub profile contains a pre-populated dirindex file, for the specific turnkey app, already. And files that have not changed here would not need to be backed up (since they are part of the base installation). But since that is not populated for internal profiles, it would be best to create an internal profile from scratch, before configuring a new app. Is that all correct?

Thank you again for all the support and work you put in on turnkey linux. I think it is such great tool and am very appreciative.

I've been a bit under the pump behind the scenes and so have fallen behind on the forums. Apologies on that.

As you've both discovered, there are no pre-compiled TKLBAM profiles. available for download. There are a few ways around the issue:

1. Create a Hub account, link to AWS and initialise tklbam. Then you can find the profile in /var/lib/tklbam (IIRC the sub dir is 'profile'). If you don't intend to use the Hub, be sure to at least downgrade to free plans (so you aren't charged).
2. Initialise with no profile, then explicitly add the desired included paths to /etc/tklbam/overrides.
3. Generate your own profile; starting with our profile source if you wish.

Obviously the first 2 are sub-optimal, but I included them for completeness. Regardless, it looks like you're both really close (acheving #3)! Please carefully re-read the TKLBAM profile's README. The important part is right near the bottom, namely:

tklbam internal create-profile --help

If you run that command (and read the help), it should be enough to get you going. But just in case it's not, here's how to generate the Nextcloud profile (assuming 'tklbam-profiles' is cloned to /turnkey/tklbam-profiles):

mkdir ~/nextcloud_profile
tklbam internal create-profile ~/nextcloud_profile /turnkey/tklbam-profiles/nextcloud
tar -cjf nextcloud_profile.tar.gz nextcloud_profile

And your Nextcloud profile should be found at ~/nextcloud_profile.tar.gz. IIRC you can use the tar.gz file directly (with tklbam-init), but if that doesn't work, unpack it first (and point tklbam-init to the directory).

To answer/address some of your initial questions/concerns Abujed:

But many of these are old and specifically the one for Nextcloud seems to point only to an outdated Nextcloud path and doesn't include the backing up of the supporting applications (apache2, mysql, php configs, etc.)

Yes they are really old, but they rarely need adjustment. They all inherit from core, which is pretty comprehensive. That's why many of them are completely empty!

Generally the only paths that you'd need to include are specific /var/lib paths (for packages you've installed from Debian - or another source that uses similar paths). Most third party stuff will go in /opt, /usr/local and/or /var/www (which are all included by default). MySQL (and PostgreSQL) DBs are auto detected and included (when they exist).

As a general rule, we don't update old paths (although we usually do eventually). The rationale is that some users may have migrated their data (from an older TurnKey appliance) so they may still have data in that location which they wish to keep. For newer users (where that path doesn't even exist) it will just be skipped.

I hope that clears things up. If either of you have further questions or concerns, please ask.

According to this post in 2014, the github repo should have the profiles that tklbam can use but they are not formatted like profiles. So I'm not sure how to use these.

I started researching tklbam about a year ago and ran into the very same questions, following an eerily similar path. I never did find a solution.

I have taken another look recently because of the upcoming EOL of buster, and the upcoming turnkey 17 releases.

Abujed did you ever find a solution to this? It would be good to know if this is indeed exclusive to turnkey hub.

Long story short; TKLBAM doesn't by default inherit from the core profile as I previously noted...

Thanks too for your kind words. :)

With such detailed replies I don't know how you ever whittle down the backlog!!!

Thank you so much!

I didn't realize that "core" profile was there at all, or its inheritance relationship... Guess I should have dug a little deeper, stumbling on that alone might have forestalled this inquiry altogether! :-D

I also hadn't picked up that you automagically (after much hard work behind the scenes) pick up the web servers and DBs.

Again, thank you very much - it's never easy working in air-gapped environments - this will help a LOT.