bhruska's picture

I've got a server that I accidently told to keep all backups on amazon.  It is not hosted on amazon, but rather on its own server.  I still want it to be backed up, but I want to delete the 100Gig of old backups.  

Can I just use the DESTROY option?  Will it then make a new backup the next time?  

Thanks,
Brian

 

Forum: 
Jeremy Davis's picture

If you destroy the backup, then you'll need to reinitialise TKLBAM (otherwise it will try to keep backing up to the destroyed record - and fail).

But if you just set "max backups" to the desired amount, then next time TKLBAM successfully completes a full backup, it will prune all the old backups. If you want to force it to run the cleanup now, simply force a new full backup:

tklbam-backup --full-backup now

Once that successfully completes, the Hub will prune the full backups back to the number of "max backups" you've set (and all the incrementals between them).

I hope that helps and all makes sense.

bhruska's picture

I reset the number of backups down to 10, and it did not prune the old ones.  I let it run for a few days.

Then I ran

tklbam-backup --full-backup now

and that didn't  work either.  The storage just got larger!  I've got 245 sessions saved.

More ideas?

Thanks,
Brian

 

Jeremy Davis's picture

Sorry if it wasn't clear, but "Max backups" only counts "full backups", not incrementals. Whilst it may not have been immediately obvious to you, that's actually noted within the Hub when you adjust "Max backups". Perhaps we need to somehow make that clearer? Do you have any suggestions on wording that may have made it clearer to you?

FWIW incremental backups aren't counted, because they form a chain; with all the previous backups, back to and including the previous full backup required to restore an incremental backup. So deleting random incremental backups within the chain (or the full backup that they ultimately rely on) essentially makes them all useless. Please also see the relevant TKLBAM FAQ entry.

So to explain further, assuming that you are using TKLBAM defaults (daily incremental, monthly full backup) then setting "max backups" to 10 will cause it to keep 10 months worth of backups. I.e. approx ~300 individual backup sessions (sessions include both full and incremental backups). So it sounds like it's working exactly as intended...! Although obviously not exactly as you were expecting...

Unless you need to keep a long history, then I find setting "max backups" to 2 works quite well. With default TKLBAM config, at most it will keep nearly 3 months worth of backups.

I.e. the day before it runs the 3rd full backup, you will have ~60 sessions, consisting of 2 old full backups (one from just under 2 months ago, one from just under 1 month ago), the ~30 incrementals between them, plus ~30 (i.e. almost a month's worth of) incrementals up until today. Tomorrow, once the new (and now third) full backup runs successfully, TKLBAM will prune the oldest full backup, plus all the incrementals depending on it. Thus leaving you with ~30 sessions, consisting of 1 old full backup, a month's worth of incrementals and today's full backup (i.e. 2 full backups, plus the relevant incremental backups).

I hope that clears things up...

bhruska's picture

So that's really confusing to me.  I don't think I changed anything from the default config, and I ended up with "unlimited" backups.  Some of my older machines I had set it to 10, but not all of them have a "runaway growth" in the amount of space its taking.  So I don't really get it in spite of your galent effort to explain!

Anyhow, the last you said was that a Maximum of 2 should work well.  And that should automatically trim down the size after a couple days.  Is that right?

Thanks,

Brian

Jeremy Davis's picture

You are right, "unlimited" is the default setting for TKLBAM backups. I.e. keep all backups forever.

Without drilling down into each of your different backups and servers, it's hard to explain what might be going on with your other backups. However, I assume that the servers are doing different things and have different traffic and that likely would explain it?!

So to answer your question, so long as you are ok with only (about) a month's worth of backups, then 2 is probably a good setting. Although if you have any doubts, you're probably better to err on the side of caution. E.g. if you later realise that something important accidentally got deleted 3 months ago, but only have a months worth of backups, you're out of luck...


So even though it's a bit complicated and the specifics regarding the different results you're seeing on different servers, hopefully I might be able to explain a little?! Let me try anyway...

Essentially a full TKLBAM backup is a diff (i.e. all the differences) between the default appliance as of when you first installed and now. Then an incremental nackup is a diff between the previous backup and now.

E.g. at one extreme, if you run a backup of a freshly installed appliance (before you've really done anything) it will be tiny. If you make few or no changes then all the backups will be tiny (incrementals will likely be a few hundred KB). Even many years worth of backups of this server (that is basically doing nothing) will be a few hundred MB at most.

At another extreme, if lots of new files are created each day, then the backups will grow exponentially. Each full backup will be much larger than the full backup before, but each incremental backup will also be quite large>

There are other complicating factors too. Diffs work much better on plain text files than on some other binary formats. In the case of some formats, that is to the point that sometimes it's actually more efficient to just save the whole file again (as if it's a new/different file).

Then there is also compression. Again text files compress really well. Sometimes as much as 10 to 1. So increases in data stored in text files, will not make huge differences to backup sizes (it all adds up over time, but not by much each backup). Whereas other file formats don't compress well (sometimes they're compressed already). Archives are a particular case in point as option whilst the filesize may not be much bigger, if the internals are quite different, then for the purposes of a backup, they're essentially a new file.

I'm not sure if I've actually helped, or just muddies the waters... If you want me to try again, then feel free to share a bit more about the specific servers, what you're usng them for and what files they contain and perhaps I can make more sense..?!

Add new comment