TurnKey Linux Virtual Appliance Library

TKLBAM: Parallel S3 uploads

In the announcement of TKL version 12, there's a mention of version 1.2 of TKKLBAM being included....and I'm particularly interested in this option:

Multipart parallel S3 uploads dramatically speed up how long it takes to upload backup archives to the cloud. Dial this up high enough and you can typically fully saturate your network connection.     http://www.turnkeylinux.org/blog/turnkey-12

In Github I found a mention of the S3-parallel-uploads flag (https://github.com/turnkeylinux/tklbam/blob/master/docs/tklbam-backup.txt), but I've tried to insert it in the overrides for TKLBAM from within Webmin, and no matter how I tweak the line I just can't get it to go - I get the following error when I try to run a simulated backup:

> tklbam-backup --simulate
Traceback (most recent call last):
File "/usr/bin/tklbam-backup", line 366, in <module>
main()
File "/usr/bin/tklbam-backup", line 152, in main
conf = Conf()
File "/usr/lib/tklbam/conf.py", line 141, in __init__
self.overrides = Limits.fromfile(self.paths.overrides)
File "/usr/lib/tklbam/conf.py", line 45, in fromfile
raise Error(`limit` + " is not a legal limit")
conf.Error: "--s3-parallel-uploads=2" is not a legal limit

I've tried it with quotes, without quotes, without the double dash, with the double dash....

In looking at the source at /usr/bin/tklbam-backup I see that it does indeed mention this variable, so I'm pretty sure I've got the latest version.

Help?  :-)

Jarrod

Jeremy's picture

Have you tried from the commandline?

I suspect that it may not be available via Webmin. Note it is not an override but an option.

I would do some testing from the commandline and when you have a number you are happy with then adjust the cron job so that it uses it all the time.

I think I'm there.....

I edited the Cron entry calling the tklbam-backup script:

OLD:  tklbam-backup --quiet

NEW: tklbam-backup --quiet --s3-parallel-uploads=4

Running a simulation, I didn't get any complaints from the system.

I don't think the users would appreciate me testing this in the middle of the work day and killing the Internet line, so I'll wait and see what happens tonight.


Looks like all is well!

Granted, we've got a faster Internet line than the last time I tried to run a backup this large....but last night's backup shoved 2.3 GB of data over to S3 in 29 minutes and some change.

And the acid test - a restore to an EC2 server - went as smoothly as I could have hoped this morning.

Thanks for the pointer to the command line, Jeremy!


Jeremy's picture

Good work! :)

Sounds like all went well then!

Post new comment

The content of this field is kept private and will not be shown publicly. If you have a Gravatar account, used to display your avatar.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <p> <span> <div> <h1> <h2> <h3> <h4> <h5> <h6> <img> <map> <area> <hr> <br> <br /> <ul> <ol> <li> <dl> <dt> <dd> <table> <tr> <td> <em> <b> <u> <i> <strong> <font> <del> <ins> <sub> <sup> <quote> <blockquote> <pre> <address> <code> <cite> <strike> <caption>

More information about formatting options

Leave this field empty. It's part of a security mechanism.
(Dear spammers: moderators are notified of all new posts. Spam is deleted immediately)