LSR's picture

Hello,

 

Newly installed turnkey appliance running in my building. No issues with giving the hub API key to the webmin interface. 

TKLBAM errors on upload to amazon and gives up after 5 tries:

Uploading s3://s3-eu-west-1.amazonaws.com/tklbam-yibysi4jdq3dnnxh/duplicity-full.20130910T151200Z.vol1.difftar.gpg to STANDARD Storage
Upload 's3://s3-eu-west-1.amazonaws.com/tklbam-yibysi4jdq3dnnxh/duplicity-full.20130910T151200Z.vol1.difftar.gpg' failed (attempt #1, reason: S3ResponseError: S3ResponseError: 417 Expectation failed 

Any ideas how to fix? Or is this an issue with EU WEST (ireland) ?

Cheers

Forum: 
Jeremy Davis's picture

Do you have a firewall between your appliance and the net? That would be the most obvious possibility...

Alexander's picture

I have the same issue with turnkey-redmine-12.1-squeeze-amd64.
For the test purpose I have install

turnkey-redmine-12.1-squeeze-i386
and
turnkey-core-12.1-squeeze-amd64

tklbam-backup produce the same S3ResponseError: 417

With new-installed

turnkey-redmine-11.1-lucid-x86

tklbam-backup/restore work fine without any problem.
ESXi 5.0.0 free hipervisor have been used in all cases with the same network environment
on the same Hub account. AWS region Tokyo (North-East Asia)

Related links:

http://www.turnkeylinux.org/forum/support/20130130/tklbam-backup...

https://github.com/turnkeylinux/tracker/issues/94

Any ideas how to fix it?

 


Jeremy Davis's picture

I just downloaded turnkey-core-12.1-squeeze-amd64.iso, installed to a fresh VM (KVM), applied by TKLBAM API key and ran a backup. It completed successfully...

--------------[ Backup Statistics ]--------------
StartTime 1379031322.00 (Fri Sep 13 00:15:22 2013)
EndTime 1379031322.08 (Fri Sep 13 00:15:22 2013)
ElapsedTime 0.08 (0.08 seconds)
SourceFiles 79
SourceFileSize 864021 (844 KB)
NewFiles 79
NewFileSize 864021 (844 KB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 79
RawDeltaSize 753429 (736 KB)
TotalDestinationSizeChange 664921 (649 KB)
Errors 0
-------------------------------------------------

So I can only assume that it is a network issue of some type - it's the only thing that makes sense... Considering that your v11.1 appliance works ok then perhaps there is some difference in the way the networking is configured between these machines (in the VMware settings). Or perhaps there is some incompatability between the networking on Debian (basis of v12.x and later) vs Ubuntu (basis of v11.x and previous) in regards to VMware. Whilst technically VMware should support it, the reality seems to be that it is not always...

To help you try and work it out, can you ping AWS servers from within the v12.x VM?

LSR's picture

I cannot ping s3-eu-west-1.amazonaws.com from any machine nor the turnkey appliance. 

We do have a firewall but the applicance is set as an always allowed machine. 


 

Alexander's picture

In my case ping of s3-ap-northeast-1.amazonaws.com address is ok for all

turnkey-redmine-12.1-squeeze-amd64
and
turnkey-redmine-12.1-squeeze-i386
and
turnkey-core-12.1-squeeze-amd64
and
turnkey-redmine-11.1-lucid-x86

root@v-redmine64 /# ping s3-ap-northeast-1.amazonaws.com
PING s3-ap-northeast-1.amazonaws.com (103.246.150.128) 56(84) bytes of data.
64 bytes from 103.246.150.128: icmp_req=1 ttl=236 time=299 ms
64 bytes from 103.246.150.128: icmp_req=2 ttl=236 time=299 ms

tklbam-backup work good only with turnkey-redmine-11.1-lucid-x86


LSR's picture

I can however ping s3-ap-northeast-1.amazonaws.com like above.

Whats wrong with s3-eu-west-1.amazonaws.com? 


 

Jeremy Davis's picture

@LSR - I just tried to ping s3-eu-west-1.amazonaws.com too and it times out for me too...  I have no idea why it's not working but perhaps it's an issue on Amazon's end? Surely it wouldn't be out this long though...?

@Alexander - I can ping s3-ap-northeast-1.amazonaws.com too. I was thinking that it may have been some sort of networking issue with Debian vs Ubuntu on VMware... But perhaps not if you can ping ok from within the server but TKLBAM fails...

I'm going to add a note to the issue you link to Alexander and encourage the devs to have a look at this... 

In the meantime all I can suggest is perhaps try reinitialsiing TKLBAM... Not sure if that will make any difference but perhaps worth a try...

LSR's picture

Is there anyway to force TKLBAM to use a different s3 server? s3-eu-west-1.amazonaws.com is still timing out and I am at a loss to a 2 week downtime.

Jeremy Davis's picture

No in the sense that there is no switch or option that I am aware of to force it to use a specific location.

But the code is all on GitHub so if you're handy with Python (or know someone who is) I'm sure you could hack it to do that...

Another thought is that you could try uninstalling and reinstalling TKLBAM (in the hope that it will realise that there is no response from that domain name...)

apt-get remove --purge tklbam
apt-get update
apt-get install tklbam
tklbam-init --force <HUB_KEY>

Although it may make no difference...

Another idea (which may not have a lot of merrit) is adjust the hosts file on your server (temporarily) to point to the desired IP. If you do that make sure that you run a full backup straight away...

LSR's picture

Purge did not sucessfully make it change to another server. It doesnt detect that its unresponsive.

Its never done a tklbam backup so I will just abandon tklbam and write a cron script to backup to local NAS. 

Ramon's picture

Hi there!

I too am having the same problem trying to backup to S3.
I have installed the latest appliance of FileServer 12.1 and I'm having the same problem.
I also installed the previous version (12.0) on another machine and still got the same problem.

I have on the same network, another appliance installed of Symfony 11.3 which TKLBAM worked fine.

All three are on the same network and all of them are VMWare's Virtual Machines.

Jeremy Davis's picture

TKLBAM v1.3 was recently released so perhaps try updating it...

apt-get update
apt-get install tklbam

Please post back with what (if any) difference that makes.

Ramon's picture

Hi Jeremy,

I just tried what you suggested and still dont work.

Jeremy Davis's picture

The update? Did it report that TKLBAM is already at the latest version? Or the TKLBAM backup (still)?

Either way...bugger!

Ramon's picture

Sorry for my short response...

The update was already at the latest version, but I tried to run anyway and didnt wok.

I am still getting the same error.

Jeremy Davis's picture

It looks like the TKLBAM v1.3 package hasn't made it to TKLv12.x (Squeeze) repo yet... TBH I can't be sure but I suspect that it will work...

If you wish to test it then I suggest that you try downloading the relevant package (from here) keep in mind that i386 refers to 32 bit and amd64 refers to 64 bit (if you get the wrong one no doubt your system will let you know when you try to install (will error)

Install with

dpkg -i <package-name>.deb

If it doesn't work, please post the error and we'll see what we can do...

Ramon's picture

I tried your suggestion and it seems to be complaining about some "pycurl-wrapper" version that I dont know how to update it.

When I ran your suggestion I've got:

(Reading database ... 22334 files and directories currently installed.)
Preparing to replace tklbam 1.2+16+g31c5ed2 (using tklbam_1.3_amd64.deb) ...
Unpacking replacement tklbam ...
dpkg: dependency problems prevent configuration of tklbam:
 tklbam depends on pycurl-wrapper (>= 1.1+2); however:
  Version of pycurl-wrapper on system is 1.1+1+gfdf654e.
dpkg: error processing tklbam (--install):
 dependency problems - leaving unconfigured
Processing triggers for man-db ...
Errors were encountered while processing:
 tklbam
Jeremy Davis's picture

Assuming you are using 32 bit version (if using 64 bit substitiute "amd64' whenever it says 'i386'...)

wget http://archive.turnkeylinux.org/debian/pool/wheezy/main/p/pycurl-wrapper/pycurl-wrapper_1.1%2b3%2bg6509b67_i386.deb
dpkg -i pycurl-wrapper_1.1+3+g6509b67_i386.deb

And then to fix the half installed new version of TKLBAM:

apt-get install -f

And now the new version of TKLBAM is installed.... Now I just hope that it resolves the issue you've been having....

[after thought] There are also new version of tklbam-squid and tklbam-squid-common... Perhaps they might be worth updating too? I'd probably test TKLBAM after installing the new version, then if that still isn't working try installing both of those tklbam-squid packages and test again... If it still doesn't work then we'll just have to throw our arms in the air and swear or something...

Ramon's picture

Hi Jeremy!

I downloaded and installed the new version of pycurl_wrapper (I'm using 64 bit by the way) and tried to run the backup again. I've got the following error:

Traceback (most recent call last):

  File "/usr/bin/tklbam-init", line 26, in 
    import hub
  File "/usr/lib/tklbam/hub.py", line 88, in 
    from pycurl_wrapper import API as _API
ImportError: No module named pycurl_wrapper
 
Then i uninstalled TKLBAM and installed again the version 1.3 that you suggested, but I still get the same error.
What could it be now?
Jeremy Davis's picture

After closer inspection it seems that Python v2.7 is the default in 13.x (Wheezy ) whereas in v12.x (Squeeze) the default is v2.6... That is why you are getting that error.... (because the newer version of pycurl_wrapper is installed to the python2.7 directory).

You can work around this error by creating a symlink to the new pycurl_wrapper like this:

ln -s /usr/lib/python2.7/dist-packages/pycurl_wrapper.py /usr/lib/python2.6/dist-packages/pycurl_wrapper.py

But TBH I'm not sure what other issues we might be creating.... But seeing as TKLBAM wasn't working before I guess we don't have anything to lose...

Ramon's picture

Hi Jeremy,

The link creation worked alright and the tklbam-backup ran again, but....

It came again to the problem at the beginning of this topic...

Lets have a look again, if you dont mind: When I run the tklbam-backup it starts listing some files and then starts displaying the following error:

 

Uploading s3://s3-sa-east-1.amazonaws.com/tklbam-lzjwc5b3n2upptik/duplicity-full.20130922T172423Z.vol1.difftar.gpg to STANDARD Storage 
Upload 's3://s3-sa-east-1.amazonaws.com/tklbam-lzjwc5b3n2upptik/duplicity-full.20130922T172423Z.vol1.difftar.gpg' failed (attempt #1, reason: S3ResponseError: S3ResponseError: 417 Expectation failed 
None) 
Uploading s3://s3-sa-east-1.amazonaws.com/tklbam-lzjwc5b3n2upptik/duplicity-full.20130922T172423Z.vol1.difftar.gpg to STANDARD Storage 
Upload 's3://s3-sa-east-1.amazonaws.com/tklbam-lzjwc5b3n2upptik/duplicity-full.20130922T172423Z.vol1.difftar.gpg' failed (attempt #2, reason: S3ResponseError: S3ResponseError: 417 Expectation failed 
None) 
Uploading s3://s3-sa-east-1.amazonaws.com/tklbam-lzjwc5b3n2upptik/duplicity-full.20130922T172423Z.vol1.difftar.gpg to STANDARD Storage 
Upload 's3://s3-sa-east-1.amazonaws.com/tklbam-lzjwc5b3n2upptik/duplicity-full.20130922T172423Z.vol1.difftar.gpg' failed (attempt #3, reason: S3ResponseError: S3ResponseError: 417 Expectation failed 
None) 
Uploading s3://s3-sa-east-1.amazonaws.com/tklbam-lzjwc5b3n2upptik/duplicity-full.20130922T172423Z.vol1.difftar.gpg to STANDARD Storage 
Upload 's3://s3-sa-east-1.amazonaws.com/tklbam-lzjwc5b3n2upptik/duplicity-full.20130922T172423Z.vol1.difftar.gpg' failed (attempt #4, reason: S3ResponseError: S3ResponseError: 417 Expectation failed 
None) 
Uploading s3://s3-sa-east-1.amazonaws.com/tklbam-lzjwc5b3n2upptik/duplicity-full.20130922T172423Z.vol1.difftar.gpg to STANDARD Storage 
Upload 's3://s3-sa-east-1.amazonaws.com/tklbam-lzjwc5b3n2upptik/duplicity-full.20130922T172423Z.vol1.difftar.gpg' failed (attempt #5, reason: S3ResponseError: S3ResponseError: 417 Expectation failed 
None) 
Giving up trying to upload s3://s3-sa-east-1.amazonaws.com/tklbam-lzjwc5b3n2upptik/duplicity-full.20130922T172423Z.vol1.difftar.gpg after 5 attempts 
Backend error detail: Traceback (most recent call last): 
File "/usr/lib/tklbam/deps/bin/duplicity", line 1405, in <module> 
with_tempdir(main) 
File "/usr/lib/tklbam/deps/bin/duplicity", line 1398, in with_tempdir 
fn() 
File "/usr/lib/tklbam/deps/bin/duplicity", line 1368, in main 
full_backup(col_stats) 
File "/usr/lib/tklbam/deps/bin/duplicity", line 513, in full_backup 
globals.backend) 
File "/usr/lib/tklbam/deps/bin/duplicity", line 412, in write_multivol 
(tdp, dest_filename, vol_num))) 
File "/usr/lib/tklbam/deps/lib/python2.6/dist-packages/duplicity/asyncscheduler.py", line 145, in schedule_task 
return self.__run_synchronously(fn, params) 
File "/usr/lib/tklbam/deps/lib/python2.6/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously 
ret = fn(*params) 
File "/usr/lib/tklbam/deps/bin/duplicity", line 411, in <lambda> 
async_waiters.append(io_scheduler.schedule_task(lambda tdp, dest_filename, vol_num: put(tdp, dest_filename, vol_num), 
File "/usr/lib/tklbam/deps/bin/duplicity", line 309, in put 
backend.put(tdp, dest_filename) 
File "/usr/lib/tklbam/deps/lib/python2.6/dist-packages/duplicity/backends/_boto_single.py", line 230, in put 
raise BackendException("Error uploading %s/%s" % (self.straight_url, remote_filename)) 
BackendException: Error uploading s3://s3-sa-east-1.amazonaws.com/tklbam-lzjwc5b3n2upptik/duplicity-full.20130922T172423Z.vol1.difftar.gpg 

BackendException: Error uploading s3://s3-sa-east-1.amazonaws.com/tklbam-lzjwc5b3n2upptik/duplicity-full.20130922T172423Z.vol1.difftar.gpg 
Traceback (most recent call last): 
File "/usr/bin/tklbam-backup", line 366, in <module> 
main() 
File "/usr/bin/tklbam-backup", line 321, in main 
b.run(opt_debug) 
File "/usr/lib/tklbam/backup.py", line 237, in run 
backup_command.run(passphrase, self.credentials, debug=debug) 
File "/usr/lib/tklbam/duplicity.py", line 78, in run 
raise Error("non-zero exitcode (%d) from backup command: %s" % (exitcode, str(self))) 
duplicity.Error: non-zero exitcode (23) from backup command: duplicity --verbosity=5 --archive-dir=/var/cache/duplicity --volsize=25 --full-if-older-than=1M --include=/TKLBAM --gpg-options=--cipher-algo=aes --include-filelist=/TKLBAM/fsdelta-olist --exclude=** --archive-dir=/var/cache/duplicity --s3-unencrypted-connection --allow-source-mismatch / s3://s3-sa-east-1.amazonaws.com/tklbam-lzjwc5b3n2upptik 

Ramon's picture

Hi,

Do you think it may be this "duplicity" thing?

I dont know what it its but it seems to have something to do with the error.

Ramon's picture

Hi Jeremy.

I found out that the erlier version of FileServer is working fine.

I installed the FileServer 11.3 Lucid version and the TKLBAM worked just fine.

There is something about the Squeeze version that's not working OK.

Jeremy Davis's picture

TBH I have no idea why you (and a couple of others) are having these issues. It doesn't make any sense to me as most users are not having any problems... There must be something unique about the circumstance. It's especially interesting that a previous TKLBAM version owrks ok (I assume under the same circumstance).

Have you tried the new TKL v13.0 appliances? AFAIK they have the new version of TKLBAM preinstalled...

Alexander's picture

1. TKLBAM from 12.1 & 13.0 versions of the Turnkey Linux produce Error: 417 Expectation failed while backup to AWS, but 11.1 work without the error.

2. Adding option

ignore_expect_100 on

in /etc/squid/squid.conf of my proxy resolve the problem for me.

3. May be the issue is the result of RFC 2616 violation:

http://www.squid-cache.org/Doc/config/ignore_expect_100/

http://www.ietf.org/rfc/rfc2616.txt


Add new comment