Rog_Enk's picture

Hi.

 

I have 77GB of backups that I am unable to delete, have tried FireFox and Chrome to no avail.

Delete job is accepted but it times out.

Would really like to remove this before I get billed again for it!

Is there an alternative method to remove these?

Thanks in advance :)

Forum: 
Jeremy Davis's picture

I thought there may have been a commandline option but it seems not. I suggest that you try using the Feedback button (blue button on the left when logged into the Hub) and see if the devs can help you out. Other than that I guess all you can do is keep trying, perhaps try from a different computer altogether if you can?

Rog_Enk's picture

Thought so.

I had tried the feedback, will try again and see if I get a response this time.

Thanks for responding :)


Alon Swartz's picture

Insight as to what happening:

  • Each TKLBAM backup is stored in its own bucket, with each session split into multiple files (ie. keys).
  • Amazon doesn't provide a way to delete a bucket and all keys at once, instead you need to recursively issue a delete on each key, and finally delete the bucket.
  • The browser session is timing out, which is causing the recursive delete to halt, but, the keys are being deleted even though it is not represented as such in the Hub.

The workaround is simple, just continue to click delete until it succeeds - it will eventually.

I know this is not a solution, but it will work until we fix the bug - move the deletion logic to background task so timeouts won't be an issue.

Sorry about this, but hopefully the workaround will help until we implement the fix.

Liraz Siri's picture

The problem seems to be that the Hub is timing out when it tries to delete large backup sets. We're going to bump the priority for this. I'm pretty sure we can prevent this from happening by adding more failure redundancy and robustness to the code. Maybe deleting backup sets asynchronously instead of synchronously within the session.

In the meantime, here's a workaround that involves peeking under the hood and messing by hand with a few raw TKLBAM internals. It's not pretty folks, but if all else fails this will work.

First we set up access to Duplicity, which is the back-end component TKLBAM uses to store your backup archives to S3:

    root@core ~# tklbam-backup --simulate --debug
CREATING /TKLBAM
FULL UNCOMPRESSED FOOTPRINT: 2.40 MB in 233 files

# duplicity --verbosity=5 --archive-dir=/var/cache/duplicity cleanup --force s3://s3.amazonaws.com/tklbam-redogwr32hmweqpt

# PASSPHRASE=$(cat /var/lib/tklbam/secret) duplicity --verbosity=5 --archive-dir=/var/cache/duplicity --volsize=25 --full-if-older-than=1M --include=/TKLBAM --gpg-options=--cipher-algo=aes --include-filelist=/TKLBAM/fsdelta-olist --exclude=** --archive-dir=/var/cache/duplicity --s3-unencrypted-connection --allow-source-mismatch --dry-run / s3://s3.amazonaws.com/tklbam-redogwr32hmweqpt

root@core ~# ADDR=s3://s3.amazonaws.com/tklbam-redogwr32hmweqpt
root@core ~# PATH=/usr/lib/tklbam/deps/bin:$PATH

A bit of explanation of what we just did. The tklbam --debug command executes a sub-shell that sets up various environment variables we need to authenticate Duplicity to AWS S3. We use --simulate together with the --debug option to prevent any Duplicity command from actually being executed.

The S3 address where our backup is stored is s3://s3.amazonaws.com/tklbam-redogwr32hmweqpt

We can get it from the last argument of the first simulated duplicity command:

# duplicity --verbosity=5 --archive-dir=/var/cache/duplicity cleanup --force s3://s3.amazonaws.com/tklbam-redogwr32hmweqpt

If the backup we wanted to delete wasn't created by this server, we'll need to get its address by executing a little bit of Python code:

root@core ~# cd /usr/lib/tklbam
root@core lib/tklbam# python
Python 2.6.6 (r266:84292, Dec 27 2010, 00:02:40)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> backup_id=11
>>>
>>> from registry import registry
>>> from hub import Backups
>>> backups = Backups(registry.sub_apikey)
>>> backup = backups.get_backup_record(backup_id)
>>> backup.address
's3://s3-eu-west-1.amazonaws.com/tklbam-redogwr32hmweqpt
>>> exit
Use exit() or Ctrl-D (i.e. EOF) to exit

So we exit the Python interpreter by pressing CTRL-D and set the ADDR:

root@core lib/tklbam# cd ~
root@core ~# ADDR=s3://s3-eu-west-1.amazonaws.com/tklbam-redogwr32hmweqpt

Now we can query the backup chain directly from our server:

root@core ~# duplicity collection-status $ADDR
Synchronizing remote metadata to local cache...
Last full backup date: Wed Feb  6 14:18:13 2013
Collection Status
-----------------
Connecting with backend: BotoBackend
Archive dir: /root/.cache/duplicity/5a17bbc9caf134ac888ea15c1b6e6fb4

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Feb  6 14:18:13 2013
Chain end time: Wed Feb  6 14:27:02 2013
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set:                            Time:      Num volumes:
                Full         Wed Feb  6 14:18:13 2013                 1
         Incremental         Wed Feb  6 14:27:02 2013                 1
-------------------------
No orphaned or incomplete backup sets found.

For safety reasons Duplicity doesn't allow you to delete ALL your backups. It likes it when you leave at least one full backup around. Assuming we want to delete absolutely everything we're going to create another "full" empty backup session:

root@core ~# mkdir /tmp/empty
root@core ~# duplicity full /tmp/empty $ADDR
root@core ~# duplicity collection-status $ADDR
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Feb  6 14:32:42 2013
Collection Status
-----------------
Connecting with backend: BotoBackend
Archive dir: /root/.cache/duplicity/5a17bbc9caf134ac888ea15c1b6e6fb4

Found 1 secondary backup chain.
Secondary chain 1 of 1:
-------------------------
Chain start time: Wed Feb  6 14:18:13 2013
Chain end time: Wed Feb  6 14:27:02 2013
Number of contained backup sets: 2
Total number of contained volumes: 2
 Type of backup set:                            Time:      Num volumes:
                Full         Wed Feb  6 14:18:13 2013                 1
         Incremental         Wed Feb  6 14:27:02 2013                 1
-------------------------


Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Feb  6 14:32:42 2013
Chain end time: Wed Feb  6 14:32:42 2013
Number of contained backup sets: 1
Total number of contained volumes: 1
 Type of backup set:                            Time:      Num volumes:
                Full         Wed Feb  6 14:32:42 2013                 1
-------------------------
No orphaned or incomplete backup sets found.

Now we ask Duplicity to delete everything before this last empty backup session:

root@core ~# duplicity --force remove-all-but-n-full 1 $ADDR
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Feb  6 14:32:42 2013
Deleting backup sets at times:
Wed Feb  6 14:18:13 2013
Wed Feb  6 14:27:02 2013
Deleting set inc Wed Feb  6 14:27:02 2013
Deleting set full Wed Feb  6 14:18:13 2013

root@core ~# duplicity collection-status $ADDR
Synchronizing remote metadata to local cache...
Last full backup date: Wed Feb  6 14:32:42 2013
Collection Status
-----------------
Connecting with backend: BotoBackend
Archive dir: /root/.cache/duplicity/5a17bbc9caf134ac888ea15c1b6e6fb4

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Feb  6 14:32:42 2013
Chain end time: Wed Feb  6 14:32:42 2013
Number of contained backup sets: 1
Total number of contained volumes: 1
 Type of backup set:                            Time:      Num volumes:
                Full         Wed Feb  6 14:32:42 2013                 1
-------------------------
No orphaned or incomplete backup sets found.

The only backup session left is the the empty backup set we created before.

The Hub will still show the older backup sessions because it didn't delete the backup itself and it assumes backups aren't deleted on the client-side (which is what we just did). In any case the backups aren't really there anymore.

Now you can click on "Delete this backup forever" to get rid of the backup record. Since there are no large backup sets to gum up the works this should work immediately.

Liraz Siri's picture

If the --debug flag throws an error that probably means you're using a version of TKLBAM older than 12. Sorry about that. You'll want to try the client-side delete from a fresh installation of TurnKey Core 12.

I added the --debug flag in tklbam 1.2, released with TurnKey 12. Unfortunately we haven't backported tklbam 1.2 it to older versions of TurnKey (e.g., 11) due to the oppressive dependency list further complicated by the move to Debian from Ubuntu.

Rick Kipfer's picture

I don't know if this helps, but I was unable to delete my backups for days until I got the suggestion to "just keep trying"...

So I did just keep trying and BOOM! they finally deleted on about the 3rd or 4th consecutive try.

This was about February 7th, 2013 or thereabouts.

Rick

Liraz Siri's picture

Yeah, each time you retried a little bit more of your backup was deleted.

Anyhow, we've figured out what the problem was and are working on a fix that will squash this once and for all.

Adi Graham's picture

Hi,

I am unable to delete my backup of 1TB.

Error Unable to delete backup record (backup in progress?). Please try again in a few minutes.

I have tried multiple times now and used the feedback button.

Set the max backups to 1 and done a new backup but the old is still there and I don't want to get charged ober $170 :(

Cheers

Adi

Jeremy Davis's picture

If you use the 'Feedback' feature in the Hub (blue button on left when logged into the Hub) then the guys can have a look.

Adi Graham's picture

Already done, I guess just have to sit back and wait now!

Jeremy Davis's picture

And assuming that you try again every now and t hen and it's still not working, then bump this thread and I'll see if I can get hold of the devs. I know that they are really busy at the mo, but they are usually pretty good at responding to help required on the Hub...

Adi Graham's picture

Appreciate the help Jeremy, I was hoping that after my new full backup of 330mb that the old one would self delete as i only want 1 full backup.

Frank Imburgio's picture

I am now able to delete od backups - but they are not deleted automatically. You have to delete the *all of the entire machine's backup* and then let the system re-create the entire chain by doing a new backup.

This thread can be closed as the devs have fixed the original problem, which was a timeout error deleting large backups.

Adi Graham's picture

I have been trying since Friday to manually delete, here is the issue below, i have also cleared down any pids which made no difference.

 

 

 

Jeremy Davis's picture

If so then perhaps another way to go is to remove the server from TKLBAM. I don't recall exactly how to do it OTTOMH but I have posted it before so a search should find it...

Also not sure if you are aware, but there are 2 types of backups, full and incremental. The incremental backups rely on all the previous incremental backups and the previous full backup (called a backup chain). You cannot remove indivdual links in the chain (i.e. AFAIK you need to remove all the more recent incremental backups first). If you set your backups to retain only one full backup and then force a full backup that might do the trick?

Adi Graham's picture

I have also created another full backup and the old one still exists.

I tried to locate your post but couldnt find it :(

Jeremy Davis's picture

At the commandline, do this:

rm -rf /var/lib/tklbam

That will remove the Hub API key (and thus disassociate your server from the Hub).

Hopefully you should then be able to delete all backups for that server. AFAIK you should then be able to re-add the key and if you set it to keep one backup only all should be well.

Adi Graham's picture

Thanks Jeremy, Support are flat out with the x64 new release but have removed the backup and tweaked the way it handles things.

Add new comment