Peter Woodall's picture

When TKLBAM does a full backup the data directories do not appear to be picked up.

I have about 4 Gb of data on the server and the last backup was 75Mb.

Unless some sort of fantastic compression is going on it looks like something is being missed.

This used to work fine.

The data is being replicated on other devices outside of the EC2 just by the way ownCloud works however it would be nice to have TKLBAM pickup the data directories.

Anyone have any ideas as to where to look for the issue?

Forum: 
Jeremy Davis's picture

Well the backup data is compressed, but I doubt it's that efficient! :) So I agree, it sounds like something isn't quite right!

There is a relatively easy way to work around that and ensure what you want is included in the backup. However, I'd like to understand a bit more as perhaps this is a TKLBAM bug (if so, most likely in the ownCloud appliance TKLBAM profile)? If so, we should fix it so it's fixed for everyone! FWIW the backup profiles are provided by the Hub, so we can update them and push them out to all users if this is indeed a bug on our behalf.

So a few questions for you:

What version of TurnKey ownCloud are you using? If you're not sure, please share the output of:

turnkey-version

Are you storing your files in the default ownCloud file location? Or have you added/adjusted the storage location on this particular ownCloud server?

When you say "this used to work fine" are you referring to this particular server? Or a previous version of TurnKey ownCloud you had running? If it's this specific server, how long has this server been running (roughly)? When was the last time it "worked fine" (roughly)?

Have you made any other modifications that you can think of at any point?

Peter Woodall's picture

Here is the version:  turnkey-owncloud-14.1-jessie-amd64

Also the backup is only 7.5 Mb which is even smaller than first thought.  This is what is reported by the Hub.

The instance has been up for 591 days so I have to retract the 'it used to work' because it may have been a previous version.

I just ran the install and didnt make any changes.

 

 

Jeremy Davis's picture

This issue was actually noted as a bug some time ago, but was fixed in v14.2.

I was under the impression that we fixed it for earlier versions too and that your server should have received the updated profile. However, I've just checked and it appears that somewhere along the line we overlooked backporting the fix to v14.1! Oops!

I'll get onto that ASAP. I've opened a new issue and pinned it to our latest release milestone to ensure that it doesn't get forgotten. In the meantime, you can either explicitly add the required location to your /etc/tklbam/overrides or reinitialise your backup forcing the v14.2 profile.

The easiest is probably just to add the required location to the TKLBAM overrides. This should do the job:

echo "/var/lib/owncloud" >> /etc/tklbam/overrides

Please let me know how you go.

Peter Woodall's picture

Thanks Jeremy, that fixed the issue and the backup picks up all data.

What is the latest version of ownCloud avaliable via TurnKey as you mentioned I am out of date?

What would be the best way to upgrade?

Thanks Again!

Jeremy Davis's picture

We made some significant changes for our v14.2 appliance, so there isn't really a super clear and easy pathway to migrate from v14.1 to v14.2, :(

We do have a ownCloud appliance doc page which provides some info and links, but unfortunately nothing quite like an easy to follow tutorial. That links to an older tutorial on migrating from the official Debian package (as installed in our v14.0 & v14.1 appliances) to the ownCloud upstream Debian repository. However, as noted in the doc page, that is no longer the recommended install method. Having said that, it may perhaps still be an option?

It's really unfortunate that we find ourselves in this situation. There are distinct advantages in using a Debian package (e.g. automated security updates that are highly unlikely to break anything). However in some cases, if the upstream software developer (in this case ownCloud) and Debian don't agree on things, it can get a bit messy.

If you want to give upgrading to the v14.2 appliance a go, then please let me know how you go. I can't offer any promises, but I'm more than happy to help as much as possible. Feel free to start a new thread for that if you wish, but posting here is fine too.

TBH, I haven't used ownCloud myself and am not really clear on the best path to upgrade. I'm really sorry that I can't provide any more pragmatic advice.

It's probably also worth mentioning that we're getting close to releasing the next release of TurnKey. It may be easier to migrate to that if you've already updated ownCloud version? Although perhaps it'd be easier to wait until we have that ready then you can go all the way to v15.0 (which will have a newer underlying OS version).

Sorry that I don't have better news, but as i say, please post back and I'll do my best to help out.

Peter Woodall's picture

Thanks for your insights.

I am going to wait until the V15.0 comes out to upgrade my two instances.

In the case of ownCloud I will spin up a new instance and then transfer the data over (it has worked that way best in the past anyway) and once stable wipe the old version.

That brings up another question about my other instance which is WordPress which has a nice clean upgrade process.

Will I be able to upgrade my existing isstance to V15 or will spinning up a new V15 WordPress instance stil be the best way to go?

Thanks again!

Jeremy Davis's picture

Holding off until we have the new v15.0 ownCloud appliance up and just transferring the files themselves sounds like a great idea. I'll certainly keep that in mind if someone else asks about it. :)

With regards to your WordPress appliance, you essentially have 2 options. Migrate your data using TKLBAM (or something else). Or, as TurnKey is based on Debian, you could do an "in place" upgrade. Both have pros and cons. Either way, assuming your WordPress is based on v14.x, it's best to wait until we have a stable v15.0 release. Please note that there isn't really any rush on that as Debian Jessie (basis of v14.x) is still receiving full support (until June this year). After that, it will move to LTS (long term support). LTS is handled by a different team, but essentially provides a similar level of support to what you would have been enjoying so far, including backported security patches.

However, if your WordPress server is still running on an earlier TurnKey version (e.g. v13.x) then I'd strongly suggest that you do a migration/upgrade ASAP. LTS security support for Debian Wheezy (the basis of v13.x) ends this month! So please get onto that ASAP!

A data migration (e.g. using TKLBAM) to v15.0 should work fairly well, although there may be some manual tweaks required. I haven't investigated at length yet, but historically as users have migrated, we've assisted them with any issues they've encountered. We've then documented as we go. Whilst the v14.x migration doc page explicitly relates migrating data from v13.x to v14.x, the same suggested workflow would still apply. If you'd like to be a part of that effort you're more than welcome to join us. OTOH, if you leave it a little while, then you'll benefit from the others that have gone before you.

The other option is to do a Debian "in place" upgrade. Debian provide comprehensive documentation on that. You'll also find many other tutorials via google. FWIW "upgrade debian jessie server to stretch" returns over 6 million results! Whilst generally that should provide a relatively straight forward pathway, please note that there is no going backwards. So once you start, you're committed and if something goes wrong, you'll need to resolve it.

You could do a trial run by restoring a TKLBAM backup to a local VM and testing there first, but there is no 100% guarantee that it will work exactly the same on your production server. Use of a snapshot may provide a better "real world" test run. The other downside of going that way, is that you won't benefit from any of the additional (non packaged) tweaks that we will be providing in v15.0 and there may still be a few manual tweaks required to get everything running smoothly.

Please note, that even though technically you could do a Debian "in place" upgrade from Jessie (basis of v14.x) to Stretch (basis of v15.x) right now, we're still ironing a few bugs in our Stretch packages so I'd suggest holding off for a little while on that (i.e. until v15.0 stable is released).

Which ever way you choose to go, I strongly suggest making backups before you proceed (TKLBAM and/or manual backups). I also recommend testing on a local VM (with a backup restore) or a snapshot before you commit yourself to the upgrade for your production server. Either way, please keep us posted on how you go and we're more than happy to give you "best effort" support with any issues you encounter.

Peter Woodall's picture

Your have given me a lot of options to choose from (and think about).

Happily both my instances are running Jessie so there is no burning need to rush forward.

Which ever method I decide on I was going to use TKLBAM to create a new EC2 instance of the Wordpress or OwnCloud server I have in Prod (I am assuming that 14.1 Jessie will restore to a new server).

I would then try out the upgrade on the clone and if I mess things up I can destroy the instance and then try again with nothing lost and no risk to Prod.

If the upgrade works I would then rejig my DNS to point to the new Instance and spin down the oold instance.

 

 

Jeremy Davis's picture

Your plan of attack sounds very sensible and reasonable to me. No need to rush into it, but getting onto it sooner rather than later is a good plan.

Re TKLBAM backup of v14.1 restored to v14.2, I would also assume that that would work perfectly smoothly, although I cannot 100% guarantee it.

As TKLBAM is primarily designed as a server backup tool first, and a migration tool second, it should always "just work" when restoring to the same server. It should also be essentially flawless when migrating to a server of exactly the same version (assuming that your backup contains everything you want/need, hence why we recommend testing restore of a backup on a new server before you need it).

Migration to a newer minor version (e.g. v14.1 -> v14.2) should generally work fine too, although there may be occasional gotchas or tweaks required from time to time. Mostly they will be related to specific appliances though, and I'm not aware of any related to the WordPress appliance. As I've already noted, migration between major versions (e.g. v14.x - > v15.0) will almost certainly require some degree of adjustment. At this point though, it seems that there won't be too many major changes (e.g. like the significant Apache and Samba config changes between v13.x & v14.x).

Add new comment