Troubleshooting TKLBAM S3 access

Generally TKLBAM "just works" but occasionally some users have issues connecting to Amazon S3.

Common issues

Common causes are firewall blocking traffic or a web proxy (common on corporate networks) blocking or mangling the traffic. For testing purposes, please ensure that you disable these (preferably) or at least create an exception for your TKL server (so it can have clear access to the internet).

Checking the log

The next step is to check the error log. I suggest that you search it for the word "error" like this:

grep error /var/log/tklbam-backup

If you are having connection errors it should return something like this:

Upload 's3://s3-eu-west-1.amazonaws.com/tklbam-eu-west-1-c992423735f1f4ee9616/24fh4htqs3t3tfju/duplicity-full.20160825T115809Z.vol1.difftar.gpg' failed (attempt #1, reason: error: [Errno 104] Connection reset by peer) 
Upload 's3://s3-eu-west-1.amazonaws.com/tklbam-eu-west-1-c992423735f1f4ee9616/24fh4htqs3t3tfju/duplicity-full.20160825T115809Z.vol1.difftar.gpg' failed (attempt #2, reason: error: [Errno 104] Connection reset by peer) 
Upload 's3://s3-eu-west-1.amazonaws.com/tklbam-eu-west-1-c992423735f1f4ee9616/24fh4htqs3t3tfju/duplicity-full.20160825T115809Z.vol1.difftar.gpg' failed (attempt #3, reason: error: [Errno 32] Broken pipe) 
Upload 's3://s3-eu-west-1.amazonaws.com/tklbam-eu-west-1-c992423735f1f4ee9616/24fh4htqs3t3tfju/duplicity-full.20160825T115809Z.vol1.difftar.gpg' failed (attempt #4, reason: error: [Errno 104] Connection reset by peer) 
Upload 's3://s3-eu-west-1.amazonaws.com/tklbam-eu-west-1-c992423735f1f4ee9616/24fh4htqs3t3tfju/duplicity-full.20160825T115809Z.vol1.difftar.gpg' failed (attempt #5, reason: error: [Errno 32] Broken pipe) 
Backend error detail: Traceback (most recent call last):

In this example the TurnKey server is obviously in Western Europe and is trying to connect to s3-eu-west-1.amazonaws.com. The errors being encountered are "Connection reset by peer" and "Broken pipe". These are errors commonly encountered when there is an issue with network connectivity and/or authorization.

Checking for network connectivity: ping

So let's check connectivity (substitute "s3-eu-west-1.amazonaws.com" for the S3 address that your TKLBAM is trying to connect to) from your TKL server to AWS S3:

ping -c 4 s3-eu-west-1.amazonaws.com

If all is well it should return something like this:

PING s3-eu-west-1.amazonaws.com (54.231.133.100) 56(84) bytes of data. 
64 bytes from s3-eu-west-1.amazonaws.com (54.231.133.100): icmp_seq=1 ttl=48 time=66.8 ms 
64 bytes from s3-eu-west-1.amazonaws.com (54.231.133.100): icmp_seq=2 ttl=48 time=104 ms 
64 bytes from s3-eu-west-1.amazonaws.com (54.231.133.100): icmp_seq=3 ttl=48 time=136 ms 
64 bytes from s3-eu-west-1.amazonaws.com (54.231.133.100): icmp_seq=4 ttl=48 time=122 ms

Obviously the times and the IP of the server you are connecting to will be different, but this shows that the TKL server can connect to s3-eu-west-1.amazonaws.com fine.

If you are not able to connect then you are having an issue with network config (firewall? internet connection?) and/or DNS resolution. Please try to resolve that issue within your network before you progress further.

Checking for S3 network connectivity: web browser

Next let's test http and https connections. These next tests assume that you have a PC (with a web browser) on the same LAN as your server and use that to find a URL and do the initial test(s). If you do not have a PC on the same LAN as your server (e.g. a remote server hosted somewhere else) then you will still need to find an appropriate S3 URL for the next step. Once we have an S3 URL that we know works, we can then check from the TKL server. But the results of this test will be irrelevant for the purposes of TKLBAM. I have a URL to test for already, but if you don't, the best way to find one is via google. E.g. google for

site:s3-eu-west-1.amazonaws.com

Then select a site that has the relevant URL. See attached screenshot (FWIW I'm just using "incognito mode" to reduce irrelevant info in my screenshots - no need for you to do that).

In your web browser click on a relevant link. I'm using "Hotspot Shield" for my example here. Ensure that the URL in the address bar is still an S3 one. Some sites may redirect to another URL. If that happens, go back to google and try a different one. Here is an example of what it should look like:

So I have just confirmed access to s3-eu-west-1.amazonaws.com using HTTPS. For a final check let's also test plain HTTP. In the address bar, remove the 's' from 'https://...' so that it is 'http://...' . Some sites may redirect back to HTTPS. If that happens, it still proves that it can connect via plain HTTP (because the server is redirecting you back to HTTPS).

If you are having errors here, and have double checked a couple of different S3 URLs (found via google) then there is obviously something wrong. Please double check firewall and network config. Otherwise, now check from within your TKL server.

Checking for S3 network connectivity: commandline (curl)

Armed with a valid (and working) S3 URL you can double check that your server can also connect properly to S3. To do that we will use curl.

Here is what I recommend. It uses curl to download the page but ignores the downloaded html (we only need to see if there is an error or not). This is the command (obviously use the working URL that you already found above).

curl http://s3-eu-west-1.amazonaws.com/tradwrds/index.html > /dev/null
If it's working as it should it should provide output something like this:
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12750  100 12750    0     0  13537      0 --:--:-- --:--:-- --:--:-- 13549
To demonstrate what it might look like if it encounters an error I mangled the URL (so I know it won't work); this is what it looks like:
curl http://s3-euz-west-1.amazonaws.com/tradwrds/index.html > /dev/null

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: s3-euz-west-1.amazonaws.com
Note the "curl: (6) Could not resolve host: s3-euz-west-1.amazonaws.com".

Again if you do not have success here, there is something wrong with your TurnKey server's network access. Assuming that that works, let's more on to deeper investigation of AWS access.

S3 access via commandline: awscli

We are again working on your TKL server. First install awscli dependancies:
apt-get update && apt-get install python-pip
Once that has completed (assuming it completes successfully) then install awscli itself:
pip install awscli

W will also need to create a user that can use the 'turnkeyhub' security profile (to double check that we can access the bucket as TKLBAM).

Browse to the AWS console IAMs user management page: https://console.aws.amazon.com/iam/home#users

Create a new user (I called mine 'tklbam-test') and click create. On the next screen click the "Show User Security Credentials" link to display the credentials info (in a drop down). It should look something like this:

tklbam-test
    Access Key ID: AKIAJ3B6NSG22LO47BUA
Secret Access Key: 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

Make sure that you copy/paste this info somewhere safe (or download the csv file) as once you leave this screen you can not access this info again (you need to recreate the keys otherwise). Obviously keep this secret and don't publish it anywhere public!

Now back on the IAMs user screen, click on the new user. Select the "Permissions" tab and click the "Attach Policy" button. You should see "turnkeyhub" listed there. If not, put it in the Filter box at the top. Once you have found it, click inside the box to the right of it (puts a tick in the box) and click "Attach Policy".

Now it's time to use those credentials we saved earlier and configure awscli. The easiest way to do the config is to simply launch awscli like this:

aws configure
Then copy/paste the relevant info into the console window. E.g.:
AWS Access Key ID [None]: AKIAJ3B6NSG22LO47BUA
AWS Secret Access Key [None]: BHKdyYXGLCERr+0RLNcKP1WWVeYt6rLR9oFHPidf
Default region name [none]: eu-west-1
Default output format [none]: 
For our purposes it doesn't matter what region we set as default, however it makes sense to use the region where TKLBAM is trying to save your backups (in this case eu-west-1). If you want to double check for valid regions, you can find the possible regions and their relevant codes here.

Now check that you can list your S3 buckets and that everything is working as it should.

aws s3 ls
Should return something like this:
2016-08-30 15:53:35 tklbam-eu-west-1-c992423735f1f4ee9616
Now see if you can look inside the bucket:
aws s3 ls tklbam-eu-west-1-c992423735f1f4ee9616
If all is well it should display something like this:
                           PRE 24fh4htqs3t3tfj/
Let's drill down again to see what is in the subdirectory:
aws s3 ls tklbam-eu-west-1-c992423735f1f4ee9616/24fh4htqs3t3tfj/
Mine looks like this:
2016-08-30 15:53:50     247045 duplicity-full-signatures.20160830T055346Z.sigtar.gpg
2016-08-30 15:53:51        218 duplicity-full.20160830T055346Z.manifest.gpg
2016-08-30 15:53:49   10168660 duplicity-full.20160830T055346Z.vol1.difftar.gpg
If you can then see all your TKLBAM volume files (like mine displays) then all is well. If you are having any permission issues while using awscli then that suggests that there is something wrong with the permissions that you have set in AWS. Most likely it is the "turnkeyhub" security profile within the IAMs "Roles" area (AWS console).