Peter's picture

Just spent 4 days patiently waiting for a 39GB to restore, and on the fourth day the webpage monitoring the restore "froze" so did the system the data was being restored to... so I crossed my fingers and rebooted.  Zip restored.  No log errors that I found even indicate a restore was underway.

Really there MUST be a better way to migrate large sites.  I have a Drupal Sute with 40GB media that is running local on a bare metal server with TK 13 (no hypervisor) , trying to migrate to a new server's TK VM with ESXI VM host.  

Frankly this is the third large migration I have tried to perform and TKBM always lets me down.  Its fine on a small data set or to backup a fresh install however when I get several gigs of data - things get awful cumbersome and I always end up resortying to some local "sneaker net" to get my data. 

Is there any plans to improve thee restore feature?  Guru's any advice for how to verify large data sets that takes days to restore IS BEING RESTORED?

Forum: 
Jeremy Davis's picture

But I think that your suggestion is a good one. Probably along with a verbosity setting which would allow users to set how verbose the output was...

And I have just put up a feature request against TKLBAM on the Issue tracker. Please check that I have maintained the spirit of what you are saying/suggesting.

But back to your problem at hand; the restore happens in stages; first - download all data; second - unarchive all data; third - restore all data. So if the previous stage has not completed; then the next stage won't start. If it did not complete the final stage then it will not look like it has restored anything (because it hasn't - although maybe it's downloaded a heap of stuff).

TKLBAM is designed to restart where it was up to if it was interrupted so perhaps that's worth a try? If you check in the config where it is saving the download data then you could see how much is there (IIRC by default it creates a new dir /TKLBAM). Also make sure that you have LOTS of free space as TKLBAM uses a minimum of 2x the size of the backup (usually lots more) plus it also needs room left over to store rollback info.

To be on the safe side; personally I'd want ~110GB free to restore a 30GB backup: 30GB for the backup download; 75GB (2.5 x) for the uncompressed data and 5GB (probably massive overkill) for the rollback data.

Logs should be in the usual spot: /var/log and IIRC restore logs should be called 'tklbam-restore'.

Dream Cutter's picture

Thanks for submitting a feature request for that.  That would be cool.  The S3 TKBM dashboard needs a few more guages to keep us occupied and less nervous about whats happing behind the scenes.  :D

 

However your insight and tips on target volume size was particulary helpful.  I figured it downloaded and imported in chunks rather than the whole backup at once.  I guess I thought since it being encrypted data rather than disk image it would be incremental approach.

I did not realize that the data may be there and it was recovering. The target VM booted to black so I didnt know what was going on.  Maybe it was attempting to recover the restore and I didnt give it enough time.  Anyways I did more research and tried other approached.  Since I am moving to eXSI VM I tried zipping the data and diung a SCP...  NO GO - To Slow.  30kbps to reimport into a eXSI VM.  eXSI doesnt like that.  So I tried a third and probably the preferred approach - to use teh VCenter Converter and thats where I am now... just connecting.

Jeremy Davis's picture

But doesn't do any restore until it has downloaded everything and uncompressed it all.

When you say "The target VM booted to black" that sounds like some whole other issue... Either some VM issue completely unrelated to TKLBAM (which perhaps was the cause of TKLBAM freezing in the first place). Or perhaps even though TKLBAM appeared to have frozen it was doing something very important (not sure what...) and being interrupted by a reboot left something a bit broken...? When I say "TKLBAM is designed to restart where it was up to if it was interrupted" I mean when you relaunch TKLBAM; not when you reboot the server...

FWIW you can also use TKLBAM to do manual backup/restore/migration and/or do the restore in stages. Have a dig through the TKLBAM man[ual] pages for options available.

Dream Cutter's picture

I agree it may have been unrelated.  Vsphere hypervisor booted to black, not the VM.  I think it was a DNS issue - VMWare is sensitive to that.  Anyway the migration via VCenter Converter _seems_ to be working faster.  At 3% of a 70GB partion already...quite a bit faster if thats an accurtate time indicator.

Dream Cutter's picture

VCenter Converter is no better - bombed out at 4% with a general systems error (undefined). I've got the servers local and on teh same lan.  Gonna pull it off line and disable all the securit/ firewall rules and just do a FTP into the VM and deal with the 40kbs speed, maybe it will be migrated by 2016. I am beginning to wonder how the big boys import large data sets into VM systems? - seems there is just no convient, reliable method to move them!  

Jeremy Davis's picture

I don't know much about VMware products, nor anything about your network, but 40mbps sounds ridiculous...

Surely there is something wrong going on there...

As a workaround, could you perhaps create a new vHDD; attach it to the server you're backing up; do a backup dump, then reattach the vHDD to the new server and do a restore...?

Dream Cutter's picture

Migration complete.  The CVenter conversion failed but after that last tip ~ I gave the aborted result an examination and low behold I find that the data did infact migrate to the target before it "failed".  It just did not restart properly - IP conflict with the LAN probably.  Adjusted settings and the VM fired right up.

One thing I did different with that last Conversion attempt from a LIVE SERVER - STOP ALL scheduled CHRON events, stop the backup and stop/disable CLAM AV.  CLAM AV was the only software package added to the original TK 13 Drupal bare metal platform.  With respect to adding data via local disk, from what I have explored adding a disk to ESXI requires a data wipe - althoug you can mount an optical.  Doesnt seem right - If it can read a CD there must be a way to attach an existing data volume w/o import.

Anyway's now its up, snapshots taken and getting it ready for its first TKBM of the VM.  Thanks for all your help.

Jeremy Davis's picture

Glad to hear that you're up and running...

As for loading new volumes on ESXi; surely there must be a better way... Although as I've never used it so I have no idea. Personally I swear by ProxmoxVE as my hypervisor of choice! :)

Add new comment