(?) Large Backups Failing / Stalling - Turnkey Filerserver

Hi Guys,

Have successfully been using Turnkey Fileserver Appliance now for over 12 months to fileserve and back up system on daily basis. Bare-metal install of Lucid appliance.

Tklbam settings are default ( full backup monthly , inc in between , and 50 M volsize)

Each night I do a local tklbam-backup to a local machine and to the remote amazon hub.

Recently backup size has got to between 12 -13 G uncompressed data footprint size. and I have noticed some remote backup failures, which I put down to internet dropouts.....

I duplicated my install on another machine to test a full back up , and while the inital backup of the system succeeded (it was only 1 MB) without all of my data, the next backup including my data fails 100% of the time, and the original machine also now fails 100 % because it has got to full backup stage as well.

What appears to be happening is that after doing the initial preparation the backup starts uploading volumes , but stalls after a certain number. I have repeated this on both my original machine and the duplicate machine with the same result.

The screen output hangs at the "Uploading ..... etc "

Analysing the output of 'ps aux' command indicates tklbam-backup still running in SL+ mode i.e in interuptable sleep mode, waiting for an event to happen.

'Netstat'  shows the amazon socket in "CLOSE_WAIT" status indicating that it has closed at its end and is waiting  for my socket to close.

After <Control-C> to kill tklbam-backup I get the following traceback ...

Uploading s3://s3-ap-southeast-1.amazonaws.com/tklbam-yfcamybrzeeisvc4/duplicity-inc.20110825T035002Z.to.20110829T061239Z.vol49.

difftar.gpg to STANDARD Storage

Processed volume 49

Uploading s3://s3-ap-southeast-1.amazonaws.com/tklbam-yfcamybrzeeisvc4/duplicity-inc.20110825T035002Z.to.20110829T061239Z.vol50.

difftar.gpg to STANDARD Storage

^CTraceback (most recent call last):

File "/usr/bin/tklbam-backup", line 266, in <module>

main()

File "/usr/bin/tklbam-backup", line 239, in main

trap = UnitedStdTrap(transparent=True)

File "/usr/lib/python2.6/dist-packages/stdtrap.py", line 266, in __init__

self.stdout_splice = self.Splicer(sys.stdout.fileno(), usepty, transparent)

File "/usr/lib/python2.6/dist-packages/stdtrap.py", line 213, in __init__

vals = self._splice(spliced_fd, usepty, transparent)

File "/usr/lib/python2.6/dist-packages/stdtrap.py", line 175, in _splice

events = poll.poll()

KeyboardInterrupt

Traceback (most recent call last):

File "/usr/bin/tklbam-backup", line 266, in <module>

main()

File "/usr/bin/tklbam-backup", line 242, in main

b.run()

File "/usr/lib/tklbam/backup.py", line 311, in run

backup_command.run(passphrase, conf.credentials)

File "/usr/lib/tklbam/duplicity.py", line 77, in run

exitcode = child.wait()

File "/usr/lib/python2.6/subprocess.py", line 1170, in wait

pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)

File "/usr/lib/python2.6/subprocess.py", line 465, in _eintr_retry_call

return func(*args)

KeyboardInterrupt

Dont know really what any of that means but hoping  the devs do.

Anyway , to me it appears that the amazon end  is timing out , or in some indefinite loop waiting for something  at my end to happen and then gives up and closes, while tklbam continues to run on my machine.

 There is no other problem at my location to indicate internet connection issues. I can transfer large files in other scenarios with no problems, and up to now tklbam has been working for me with no issues.

The log file is of no help as it just mirrors the screen output. I tried redirecting stderr to stdout but nothing , so that coupled with the fact that tklbam-backup is still running indicates  to me that no error has actually occurred in the program.

A by-product of this is that I now have 1 backup showing on my hub dashboard that still indicates "First backup in progress" and there is no way to delete it.

I would appreciate any ideas from devs/others as to where and why my problems are now occuring, hopefully the callback trace helps.

Thanks in anticipation and once again sorry about the length of the post.

Scott H.