Jarrod Igou's picture

In the announcement of TKL version 12, there's a mention of version 1.2 of TKKLBAM being included....and I'm particularly interested in this option:

Multipart parallel S3 uploads dramatically speed up how long it takes to upload backup archives to the cloud. Dial this up high enough and you can typically fully saturate your network connection.     http://www.turnkeylinux.org/blog/turnkey-12

In Github I found a mention of the S3-parallel-uploads flag (https://github.com/turnkeylinux/tklbam/blob/master/docs/tklbam-backup.txt), but I've tried to insert it in the overrides for TKLBAM from within Webmin, and no matter how I tweak the line I just can't get it to go - I get the following error when I try to run a simulated backup:

> tklbam-backup --simulate
Traceback (most recent call last):
File "/usr/bin/tklbam-backup", line 366, in <module>
main()
File "/usr/bin/tklbam-backup", line 152, in main
conf = Conf()
File "/usr/lib/tklbam/conf.py", line 141, in __init__
self.overrides = Limits.fromfile(self.paths.overrides)
File "/usr/lib/tklbam/conf.py", line 45, in fromfile
raise Error(`limit` + " is not a legal limit")
conf.Error: "--s3-parallel-uploads=2" is not a legal limit

I've tried it with quotes, without quotes, without the double dash, with the double dash....

In looking at the source at /usr/bin/tklbam-backup I see that it does indeed mention this variable, so I'm pretty sure I've got the latest version.

Help?  :-)

Jarrod

Forum: 
Jeremy Davis's picture

I suspect that it may not be available via Webmin. Note it is not an override but an option.

I would do some testing from the commandline and when you have a number you are happy with then adjust the cron job so that it uses it all the time.

Jarrod Igou's picture

I edited the Cron entry calling the tklbam-backup script:

OLD:  tklbam-backup --quiet

NEW: tklbam-backup --quiet --s3-parallel-uploads=4

Running a simulation, I didn't get any complaints from the system.

I don't think the users would appreciate me testing this in the middle of the work day and killing the Internet line, so I'll wait and see what happens tonight.


Jarrod Igou's picture

Granted, we've got a faster Internet line than the last time I tried to run a backup this large....but last night's backup shoved 2.3 GB of data over to S3 in 29 minutes and some change.

And the acid test - a restore to an EC2 server - went as smoothly as I could have hoped this morning.

Thanks for the pointer to the command line, Jeremy!


Jeremy Davis's picture

Sounds like all went well then!

Add new comment