Jack Riddle's picture

Backup ran find for a few months, but now when I try to run tklbam-backup, I get the following error:

warning: using cached profile because of a Hub error: error(77, '')
error: error(77, '')

I work with linux out of necessity and infrequently, so troubleshooting this is not easy for me. Anyone have some guidance on what I should try?

 

Forum: 
Jeremy Davis's picture

Ok, so the error suggests that the issue is somehow related to your servers connection to the Hub. Perhaps there is something wrong with your Hub account? Or perhaps something has gone wrong on your server? If your server is running locally, then perhaps the issue lays somewhere in between (e.g. your internet connection or local network)?

So I would suggest that you try logging into your Hub account and see if there is anything obvious there. Please feel free to ping me there via the "Support" link towards the left in the top bar, or the blue chat icon in the bottom right corner. I can then have a look from our end too and see if there is anything that jumps out.

To troubleshoot your server, can you please give me the output of the following commands:

turnkey-version
apt policy tklbam
ping -c4 hub.turnkeylinux.org

If you server is running locally, then I'd suggest that you investigate any local networking changes. E.g. a new ISP or changed internet plan; a new or changed proxy server to connect to the internet; a new or changed firewall setup; etc.

Jack Riddle's picture

I don't see any issues in the hub. There have been no changes to our firewall or internal environment, but our bandwidth was recently upgraded.  However, this was handled by our ISP on their own equipment--no changes were made to our environment. 

I can see the turnkey server hit our firewall when I try to run TKLBAM-Backup, and the firewall indicates the traffic is allowed:

Here are the results from the requested commands:

 

[root@wordpress ~]# tklbam-backup
warning: using cached profile because of a Hub error: error(77, '')
error: error(77, '')
[root@wordpress ~]# turnkey-version
turnkey-wordpress-15.3-stretch-amd64
[root@wordpress ~]# apt policy tklbam

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

tklbam:
  Installed: 1.4.1+37+g8117cd6
  Candidate: 1.4.1+37+g8117cd6
  Version table:
 *** 1.4.1+37+g8117cd6 999
        999 http://archive.turnkeylinux.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status
[root@wordpress ~]# ping -c4 hub.turnkeylinux.org
PING hub.turnkeylinux.org (23.21.244.168) 56(84) bytes of data.
64 bytes from ec2-23-21-244-168.compute-1.amazonaws.com (23.21.244.168): icmp_seq=1 ttl=49 time=30.8 ms
64 bytes from ec2-23-21-244-168.compute-1.amazonaws.com (23.21.244.168): icmp_seq=2 ttl=49 time=30.7 ms
64 bytes from ec2-23-21-244-168.compute-1.amazonaws.com (23.21.244.168): icmp_seq=3 ttl=49 time=30.8 ms
64 bytes from ec2-23-21-244-168.compute-1.amazonaws.com (23.21.244.168): icmp_seq=4 ttl=49 time=30.6 ms

--- hub.turnkeylinux.org ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 30.693/30.782/30.863/0.061 ms

 

Jeremy Davis's picture

Considering that you have the latest TKLBAM, everything looks ok in your Hub account, ping is responding as expected, and that I can't reproduce this issue in a local VM from my end (here in Australia), it is almost certainly something within your VM and/or your environment (e.g. host system if running as a VM; networking within your LAN and/or infrastructure including internet connection). To break it down a bit further, my guess is one of:

  • Some non-obvious change to do with the new ISP plan. Whilst rare, there is precedent for this and we've seen it before; the ISP denied any changes, but changing ISP completely resolved the issue. This seems somewhat unlikely given your notes, but can't be completely ruled out considering that fact that ISP changes coincided with this issue occurring.
  • Some local issue such as corruption of some sort, or lack of free disk space (if a VM, perhaps on the host?) - which is for some reason giving an error message that isn't related to the real cause.

So unless connecting to the internet via an alternate path is an easy option and/or you want to contact your ISP and explicitly ask them about any changes/difference that may affect encrypted traffic, let's put the ISP possibility aside. (Also FWIW even if you do contact your ISP they may deny any changes regardless).

So to focus on potential issues related to your local TurnKey appliance, assuming that it's a VM, please double check the host system has plenty of free space. If your VM has a dynamically assigned disk, if the host runs out of free space (or is very close), then the guest may act as if it has also run out of space despite showing that it still has free space. I'd also suggest checking system logs on the host to

Assuming that's ok, within your TurnKey appliance, check the available disk space like this:

df -h

Look for the line that is "Mounted on" "/" (it should either be '/dev/mapper/turnkey-root" or "/dev/sda1"). Then double check how much is "Avail" (available). E.g. here's the output of a VM I have handy:

Filesystem                Size  Used Avail Use% Mounted on
udev                      2.9G     0  2.9G   0% /dev
tmpfs                     597M   23M  574M   4% /run
/dev/mapper/turnkey-root   12G  3.5G  7.5G  32% /
tmpfs                     3.0G     0  3.0G   0% /dev/shm
tmpfs                     5.0M     0  5.0M   0% /run/lock
tmpfs                     3.0G     0  3.0G   0% /sys/fs/cgroup

Mine has a root filesystem of 12GB, with 3.5GB used and 7.5GB free. If yours has less than a few hundred MB free that is quite likely the cause. If you're unsure, please feel fre to post your output.

If everything is still looking ok for now, you could try manually clearing the caches. I'm not sure that this will help, but perhaps? Do that like this:

rm -r /var/cache/tklbam/*
rm -r /var/cache/duplicity/*

Then retry running 'tklbam-backup'.

If you still get the same issue, then the next thing to check is to re-initialise your server. This will test to see if your current credentials and config are corrupted. We'll keep the current data just in case that doesn't resolve this issue. In an ideal situation, you would have enabled as least "Backup Standard" (@$10/month) so that you can test creation of a new backup (I note that you're currently on "Backup Free). But I'm pretty sure that this process should be able to test re-initialisation without upgrading to a paid plan (even though you won't be able to fully test creation of and uploading) a backup. SO here's what you'll need to do:

mv /var/lib/tklbam /var/lib/tklbam.orig
tklbam-init YOUR_HUB_API_KEY

That should allow you to re-initialise your server as if it was a new server. It should succeed. If it does, you can then move on to test creation of a backup (this step will fail if you don't upgrade to a paid backup plan; but the error should be different to your previous error - it should explicitly note that it can't upload). So again try this:

tklbam-backup

If you get the same error as before and/or the tklbam-init step above gives a similar error message then it seems most likely to me that it must be an issue with your environment (e.g. LAN and/or internet connection). I guess you could test creating a new VM and checking to see if that can connect and if it generates the same errors or not, but I'm not sure how much that will help you diagnose. It would probably require capturing network traffic and comparing to traffic from a working system.

Jeremy Davis's picture

The logs that I have access to aren't particularly detailed, but the only notes regarding your Hub account that I can see are when you set the account up.

Jack Riddle's picture

I had other priorities as school started back that limited my ability to work on this further. Now, I'm back at it.

First, I checked space. All my values equal or exceed your own example. The host has plenty fo free space.  

Second, I cloned the VM and connected it directly to a separate internet connect--a cable modem in bridge mode--so there wasn't a firewall or anything that could interfere. No change.

Third, I attempted the re-initialise steps.  After running tklbam-init with my hub key, I get the "Generated backup encryption key" message, and then "error: error(77, ' ')"  

Running the command again returns the same error with the encryption key message.

t

My ultimate plan for getting this to work again is to migrate to a new Turnkey appliance. If there is another easy way to do that other than TKLBAM, I can put this to bed and do it another way since this server will go bye bye.

Jeremy Davis's picture

I don't understand why this is occurring?! It sounds like you have pretty much eliminated all the possibilities that I can think of?!

I've dug into the TKLBAM code a bit and given the error message that you have reported, it still appears to be related to communication with the Hub!? FWIW, line 185 is the only place in tklbam-init where it can just return a simple "error: ERROR_MESSAGE" (without any further info).

Following that thread, hub.Backups.get_sub_apikey(apikey) seems to be what's rasing the error. Looking closer at that though, it appears to be a fairly straight forward call the the Hub API. And the error that is being returned, is not one that is expected or accounted for. The "expected" possible errors all appear to be clearly documented. So I think that we can assume that it's not one of them

So my guess at this point is that either there is (still) something between you and the Hub that is mangling the traffic somewhere/somehow. Or there is something broken in your Hub account, that is not giving us a clear and/or expected error message.

So at this point, I see 2 paths forward:

  1. I provide you with a hacked file so that we'll get a more expressive error message (ideally a full stacktrace) so we can get a clearer idea of what is going on.
  2. We continue to try to "brute force" it. We're running out of options here, but the next step in this path would likely be to try reinstalling TKLBAM.
  3. Change tack completely and just try to get your WordPress data out and migrate to a new TurnKey server.

Option 1 - hack on TKLBAM

This won't actually fix the issue (and may still lead to a dead end). But it will hopefully give me more insight into what is actually wrong. You can see the changes that I've made here. And here is the command to run to download it and overwrite the default file:

FILE=cmd_init.py
URL=https://raw.githubusercontent.com/JedMeister/tklbam/troubleshooting/$FILE
LOCAL=/usr/lib/tklbam/$FILE
wget $URL -O $LOCAL
# also remove the compiled 'cmd_init.pyc' file
rm ${LOCAL}c

Then rerun the previous command and post the output.

Option 2 - Continue "brute force" approach

The next step in this approach is to just reinstall TKLBAM and cross your fingers:

apt-get update
apt-get install --reinstall tklbam

It may also be worth reinstalling it's dependencies, but the only one that should make a difference at this point is 'pycurl-wrapper':

apt-get install --reinstall pycurl-wrapper

Option 3 - Collect and migrate data via some other method

Essentially all you need is a copy of the 'wordpress' database and the relevant files (should all be in /var/www/wordpress).

If you have some idea of what you're doing with WrodPress and Linux, then you could just manually do that yourself, zip it all up and copy it across to a new server with SFTP or Rsync.

If you aren't so confident, then I suggest that either you do a bit of googling on how to do that (you should find plenty of tutorials). TurnKey is based on Debian and this is something that is generic enough that even tutorials based on other Linux distros would probably "just work" too.

Another option is to look for a backup WordPress plugin. That will take care of the heavy lifting for you and you can probably just install it on the new TurnKey server too and restore your backup using the same plugin.


Regardless of which path you take. Please let me know how you go and don't hesitate to ask if you have further questions/concerns/errors and I'll do my best to help out.

Jack Riddle's picture

For option 1, I did as requested and got the following output:
 

tklbam-init BXXXXXXXXXXXXXXXXA
Traceback (most recent call last):
  File "/usr/bin/tklbam-init", line 212, in 
    main()
  File "/usr/bin/tklbam-init", line 185, in main
    sub_apikey = hub.Backups.get_sub_apikey(apikey)
  File "/usr/lib/tklbam/hub.py", line 230, in get_sub_apikey
    response = API().request('GET', cls.API_URL + 'subkey/', {'apikey': apikey})
  File "/usr/lib/tklbam/hub.py", line 126, in request
    raise APIError(e.code, e.name, e.description)
hub.APIError: error(77, '')

For option 2, updated apt-get and ran the reinstall.  Same error message.

I'll persue option 3 in the meantime.

Jeremy Davis's picture

Thanks for that. After looking more closely at the code, I actually think that the root issue is outside TKLBAM; perhaps something not quite right (perhaps even an undocumented/missing dependency?) in python-pycurl or pycurl-wrapper.

To continue down the Option 1 path, and extend the stacktrace (and perhaps confirm my suspicions) I have another file to update temporarily for troubleshooting:

FILE=hub.py
URL=https://raw.githubusercontent.com/JedMeister/tklbam/troubleshooting/$FILE
LOCAL=/usr/lib/tklbam/$FILE
wget $URL -O $LOCAL
# also remove the compiled 'hub.pyc' file
rm ${LOCAL}c

Again if you wish to checkout what I changed you can see it here. FWIW, similar to last time, I commented out error handling code to allow more verbose stacktrace.

IMPORTANT - Please be sure to double check that your Hub API key is not included in the output that you post. FWIW it shouldn't this time, but if we continue I'll possibly include some "print" statements so we can see the data that is being passed (to ensure that it's valid). When that happens there is risk that your API key will be in the output. If it is, please replace it with a string that make it clear, e.g. MY_HUB_API_KEY or similar.

I was also think that it might be worth reinstalling some other packages (i.e. a continuation of "Option 2"), but I'm not sure of the value there, or even exactly which packages are more likely to be causing the issue, so I'll just leave that be for now...

Jack Riddle's picture

Here is the output with the 2nd file applied.  (First time I ran it, there was no output. I assume this is due to the tklbam reinstall. I reapplied the first file and then the second, and ran the init command and got this output.)
 

Traceback (most recent call last):
  File "/usr/bin/tklbam-init", line 212, in 
    main()
  File "/usr/bin/tklbam-init", line 185, in main
    sub_apikey = hub.Backups.get_sub_apikey(apikey)
  File "/usr/lib/tklbam/hub.py", line 230, in get_sub_apikey
    response = API().request('GET', cls.API_URL + 'subkey/', {'apikey': apikey})
  File "/usr/lib/tklbam/hub.py", line 117, in request
    return _API.request(self, method, url, attrs, headers)
  File "/usr/lib/python2.7/dist-packages/pycurl_wrapper.py", line 187, in request
    raise self.Error(self.ERROR, "exception", e.__class__.__name__ + `e.args`)
pycurl_wrapper.Error: 500 - exception (error(77, ''))

 

Add new comment