TurnKey Linux Virtual Appliance Library

tklbam-restore: takes forever

c.boulanger's picture
Hi, I just had the special moment where TKLBAM comes to the rescue - my home directory with all the data was erased for whatever fricking reason... now I simply do
tklbam-restore --limits="/home" 10
and I see it downloads stuff into /tmp/tklbam... - actually all files are there. But nothing has happened for a long time now, the process is still running...
tail -f /var/log/tklbam-restore 
gives me
Restoring duplicity archive from s3://s3-eu-west-1.amazonaws.com/tklbam-xxx
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Wed Oct  5 06:26:28 2011
but nothing else. I started the process 30 minutes ago - is it normal it takes so much time?
Christian's picture

It's still running, 50

It's still running, 50 Minutes now ;-( There are two tklbam-restore processes when I do `px ax`. How can I tell if it is still downloading the backup or if it is stuck?



Christian's picture

Ok, seems to have been a

Ok, seems to have been a problem associated with "no space left on device" -> my virtual harddrive was too small. Maybe a check for disk capacity could be worked into the tklbam-restore script, to give the user a warning if there is not enough space left for the restore?



Jeremy Davis's picture

That sounds like a good idea!

Especially when the default VM image size is only 20GB. It doesn't take long these days to fill that up (especially a Fileserver).

Ideally it could calculate the required space and check to see that there is enough room and let the user know if there isn't. Even if it gave an error which let the user know why it wasn't working would be better (although obviously better to know before you've downloaded a heap of stuff than after).

I have just logged a bug on the TKL bug tracker here.

Christian's picture

Thanks & Suggestion

Thanks Jeremy.

tklbam-restore really saved my life this time. Everything is up and running again. Thanks so much for your work, Alon & Liraz! Hurray!

What I didn't expect was that it did take several hours to assemble the backup. Another suggestion would be that the restore script should do either:

  • display progress ("x of y MB restored" / "x% restored", "x minutes remaining")
  • or if computing a detailed progress is not possible/too complex, at least provide a rough estimate ("x GB of data will be restored, this can take several minutes/hours/days").

This way, users like me aren't anxiously glued to the screen waiting for the script to finish when it will really take hours.



Liraz Siri's picture

How long restoring takes sort

How long restoring takes sort of depends on the size of your backup and where you are restoring to, transfer speeds from S3 and all that. Regarding improving the progress report I have that on my todo list for the next update. Most of that time is spent in Duplicity. I'll see if I can hack it to do what we want.

Another idea is to provide an email notification hook so you get an e-mail/SMS once the restore is done.

Christian's picture

E-Mail-Notification ...

... sounds like a good idea. When my system broke down, it was fortunately late in the evening, so I could let it run overnight. But in a scenario where a busy website has to be restored while people are waiting for it to go back online, it will be good to be able to do other things and immediately react once it is finished. And thanks for adding some sort of progress indicator.

Thanks again for all of your excellent work.

Liraz Siri's picture

There's a workaround for the

There's a workaround for the running out of space on /tmp issue:

mkdir -p /mnt/tmp
mount --bind /mnt/tmp /tmp

That redirects /tmp to instance-storage which is much larger than the VM's root image. We're looking into fixing this at the Hub level so this workaround should soon be unnecessary.

Jeremy Davis's picture

Out of curiosity... how does that work?

I don't understand!?! (I'm quite possibly showing the extent of my ignorance with this post, but here goes...)

I understand the commands and what they do (perhaps!) ie make a tmp directrory in /mnt and bind the /tmp directory to /mnt/tmp (so it uses /mnt/tmp rather than /tmp). But where does the extra space come from (unless you have additional space available via an extra vmdk or something which you then mount)? Or is it because the /tmp folder size is artificially limited (ie not by filesystem space but some other factor) and therefore that's why it runs out? Wouldn't it still occur (without a workaround other than to attach additional storage) if say you were restoring a 19GB backup to a 20GB VM?

I currently have TKL Core v11.2 running under VBox and mount doesn't report anything pre-mounted on /mnt and /tmp doesn't seem to be mounted as tmpfs (as I would normally expect with an Ubuntu system). Is this because of LVM? Is that why all this doesn't make any sense to me? Am I asking too many questions? :)

Liraz Siri's picture

No no, you're absolutely

No no, you're absolutely right. My bad. I thought you were using TurnKey on Amazon EC2 where several hundred GBs of instance storage are mounted to /mnt. On VBox this obviously wouldn't work. You need to extend the filesystem with LVM.

Jeremy Davis's picture


So I didn't make a fool of myself after all! Nice :)

Guest's picture

10 Gb Restore job

I have a 10 Gb restore job that has taken over 24 hours and is still restoring. How long should a restore of this size take?

Liraz Siri's picture

Added fix to Hub that mount --binds /tmp to instance storage

You don't need to mount --bind /tmp to instance storage any more. The Hub now does this for you.

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)