Forum archive
lxc tklbam-backup -> tklbam-restore'd in AWS results in broken system
I am trying to use my tklbam backups made in proxmox lxc instances into new EC2 machines.
The restored system gets essentially broken by the restore. Things like resolvconf for example gets broken. The /run/resolvconf dir is missing and /etc/resolv.conf becomes a real file (from the backup origin).
It seems, and makes sense, that the init (systemd) process is quite different in an lxc rather than in a real system running it's own kernel (VM or baremetal). The restore basically changes "things" in /etc and which packages are installed and this basically breaks the system at a very low level (inittab, systemd, etc.)
Are there recipes or excludes to apply to the backup source perhaps which might alleviate the problem?
I am currently testing this addition in my source system /etc/tklbam/overrides:
# These are only necessary for the lxc backup to VM restore -/etc/* /etc/apache2 /etc/default/locale /etc/cron.daily/myapp /etc/init.d/myapp /etc/letsencrypt /etc/logrotate.d/myapp /etc/postfix/main.cf /etc/rc*/*myapp /etc/ssh /etc/tklbam/overrides /etc/locale.gen /etc/mailname /etc/timezone
*replace myapp with the real app name ;)
I am using a turnkey-postgresql-14.0 as source (kept up to date) and a turnkey-postgresql-14.2 in AWS.
Replies (5)
Awesome thanks Jeff
Reply 1Thanks for the response!
Reply 2I ended up just doing what I mentionned. That is ignoring /etc in /etc/tklbam/overrides, then adding back only the few files which actually need to change. I also did the restore of my 14.0 backups in a running 15.0 instance.
The extra packages installed from the lxc backup don't seem to break things without the /etc files.
For a 14.0 lxc to 15.0 VM migration, i.e. only done once, I will probably opt to pre-install the extra packages the instance needs in the target (very few, and easily shown/copy-pasted in /TKLBAM/newpkgs from a tklbam-backup --simulate run), then restoring my thinned (-/etc/*) backup with --skip-packages.
Since my app is in /opt, that is it is not upgraded in the migration from 14 to 15, it just works after the restore.
Hope this helps someone! ;)
Thanks for reporting
Reply 3FWIW, I've opened a bug report on our issue tracker.
Thanks for reporting this and providing info on what you're trying so far.
First up, apologies that I didn't notice earlier that your post got flagged as spam. I have fixed that and also added your user account to the "contributors" group so that you should be able to skip most of the spam filters in future.
Although it's likely no comfort ATM, things should be at least a little better in v15.0 as LXC/Proxmox images and other builds all use the same init system now. We were forced to continue to use Sysvinit in the v14.x LXC builds because of a bug in SystemD in the version provided by Debian Jessie.
I'm not really sure why resolv.conf is being backed up as a file. I recall that that was a bug in TKLBAM at one point in the past (symlinks were being restored as files). But I was under the impression that that was resolved quite some time ago (prior to v14.0). However, perhaps there is some context in LXC that is unaccounted for?
Apart from your modifications to the over-rides, one thing that also may help is if you skip packages on the restore. You can do that via the --skip-packages. Unfortunately that will mean that any additional packages (beyond the defaults that TurnKey is installed with) which you have installed will not be re-installed on restore.
Depending on your planned usage, I can think of a couple of ways to work around that.
The easiest would probably be to just explicitly install your required set of packages via a restore hook script. If it's a "one off" restore, or your setup is pretty stable, that may be sufficient?
Although if you planned to use your backup as a way to migrate data between a local dev server and a remote production server, then that may not be ideal. You'd need to manually update that anytime you change your required package set.
Possibly a better way would be to run a pre-restore script to adjust the relevant package info from tklbam records and remove the offending packages (IIRC they were systemd-shim and sysv-init) backup script. Unfortunately, I'm not familiar enough with the inner workings of tklbam (and the order that stuff happens) to be 100% sure whether that would work. But I imagine that it should be possible. Even if it's not quite possible as I hope, perhaps you could skip-packages, but have a hook script read the tklbam package list and install all but the problematic packages.
As a bit of additional background info, all appliances inherit their backup profile from core. Many applainces (including postgresql) don't provide anything additional over and above that default profile. So once we can work this out, it should apply across all LXC/Proxmox appliances.
Thanks for the update Jeff
Reply 4I'm pretty sure that the user management is done as a separate process - which would explain /etc/passwd changing despite excluding /etc from the backup. Perhaps the moving of the sshd user's home is also related to that?
I imagine that most TKLBAM users would want the user accounts included/migrated as part of the backup. However, it would be nice to have more granular control over the migration of user accounts if/when desired. I've added that as a wishlist feature request. Not sure if/when it'll get implemented, but would be a cool addition IMO.
Using only lxc instances, I
Reply 5Using only lxc instances, I examined a before and after of the tklbam-restore --skip-packages --limit='-/etc' of a 14.0 backup to 15.0.
To be completely clear, the before is tk redmine 15.0 on lxc, the after is this same system, after a tklbam-restore -skip-packages --limit='-/etc' 37 (where the backup was made on tk redmine 14.0 using "tklbam-backup --skip-packages" and "-/etc" is excluded. (not "-/etc/*" as in my previous post).
The only unexplained change I see in /etc after the restore is /etc/passwd that gets modified, the sshd user got it's home dir modified from /run/sshd to /var/run/sshd ... ? Of course, in the end, /var/run points to /run so not a big issue... just strange.
Glad to hear that you managed to work around it for now.
When I get a chance, I'll have a closer look at that and see if we can improve the default usage of TKLBAM when transferring to/from LXC builds. I don't really have the spare cycles to look much closer ATM, but there's an issue on the tracker so it won't get forgotten.