Jack Riddle's picture

Backup ran find for a few months, but now when I try to run tklbam-backup, I get the following error:

warning: using cached profile because of a Hub error: error(77, '')
error: error(77, '')

I work with linux out of necessity and infrequently, so troubleshooting this is not easy for me. Anyone have some guidance on what I should try?

 

Forum: 
Jeremy Davis's picture

Ok, so the error suggests that the issue is somehow related to your servers connection to the Hub. Perhaps there is something wrong with your Hub account? Or perhaps something has gone wrong on your server? If your server is running locally, then perhaps the issue lays somewhere in between (e.g. your internet connection or local network)?

So I would suggest that you try logging into your Hub account and see if there is anything obvious there. Please feel free to ping me there via the "Support" link towards the left in the top bar, or the blue chat icon in the bottom right corner. I can then have a look from our end too and see if there is anything that jumps out.

To troubleshoot your server, can you please give me the output of the following commands:

turnkey-version
apt policy tklbam
ping -c4 hub.turnkeylinux.org

If you server is running locally, then I'd suggest that you investigate any local networking changes. E.g. a new ISP or changed internet plan; a new or changed proxy server to connect to the internet; a new or changed firewall setup; etc.

Jack Riddle's picture

I don't see any issues in the hub. There have been no changes to our firewall or internal environment, but our bandwidth was recently upgraded.  However, this was handled by our ISP on their own equipment--no changes were made to our environment. 

I can see the turnkey server hit our firewall when I try to run TKLBAM-Backup, and the firewall indicates the traffic is allowed:

Here are the results from the requested commands:

 

[root@wordpress ~]# tklbam-backup
warning: using cached profile because of a Hub error: error(77, '')
error: error(77, '')
[root@wordpress ~]# turnkey-version
turnkey-wordpress-15.3-stretch-amd64
[root@wordpress ~]# apt policy tklbam

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

tklbam:
  Installed: 1.4.1+37+g8117cd6
  Candidate: 1.4.1+37+g8117cd6
  Version table:
 *** 1.4.1+37+g8117cd6 999
        999 http://archive.turnkeylinux.org/debian stretch/main amd64 Packages
        100 /var/lib/dpkg/status
[root@wordpress ~]# ping -c4 hub.turnkeylinux.org
PING hub.turnkeylinux.org (23.21.244.168) 56(84) bytes of data.
64 bytes from ec2-23-21-244-168.compute-1.amazonaws.com (23.21.244.168): icmp_seq=1 ttl=49 time=30.8 ms
64 bytes from ec2-23-21-244-168.compute-1.amazonaws.com (23.21.244.168): icmp_seq=2 ttl=49 time=30.7 ms
64 bytes from ec2-23-21-244-168.compute-1.amazonaws.com (23.21.244.168): icmp_seq=3 ttl=49 time=30.8 ms
64 bytes from ec2-23-21-244-168.compute-1.amazonaws.com (23.21.244.168): icmp_seq=4 ttl=49 time=30.6 ms

--- hub.turnkeylinux.org ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 30.693/30.782/30.863/0.061 ms

 

Jeremy Davis's picture

Considering that you have the latest TKLBAM, everything looks ok in your Hub account, ping is responding as expected, and that I can't reproduce this issue in a local VM from my end (here in Australia), it is almost certainly something within your VM and/or your environment (e.g. host system if running as a VM; networking within your LAN and/or infrastructure including internet connection). To break it down a bit further, my guess is one of:

  • Some non-obvious change to do with the new ISP plan. Whilst rare, there is precedent for this and we've seen it before; the ISP denied any changes, but changing ISP completely resolved the issue. This seems somewhat unlikely given your notes, but can't be completely ruled out considering that fact that ISP changes coincided with this issue occurring.
  • Some local issue such as corruption of some sort, or lack of free disk space (if a VM, perhaps on the host?) - which is for some reason giving an error message that isn't related to the real cause.

So unless connecting to the internet via an alternate path is an easy option and/or you want to contact your ISP and explicitly ask them about any changes/difference that may affect encrypted traffic, let's put the ISP possibility aside. (Also FWIW even if you do contact your ISP they may deny any changes regardless).

So to focus on potential issues related to your local TurnKey appliance, assuming that it's a VM, please double check the host system has plenty of free space. If your VM has a dynamically assigned disk, if the host runs out of free space (or is very close), then the guest may act as if it has also run out of space despite showing that it still has free space. I'd also suggest checking system logs on the host to

Assuming that's ok, within your TurnKey appliance, check the available disk space like this:

df -h

Look for the line that is "Mounted on" "/" (it should either be '/dev/mapper/turnkey-root" or "/dev/sda1"). Then double check how much is "Avail" (available). E.g. here's the output of a VM I have handy:

Filesystem                Size  Used Avail Use% Mounted on
udev                      2.9G     0  2.9G   0% /dev
tmpfs                     597M   23M  574M   4% /run
/dev/mapper/turnkey-root   12G  3.5G  7.5G  32% /
tmpfs                     3.0G     0  3.0G   0% /dev/shm
tmpfs                     5.0M     0  5.0M   0% /run/lock
tmpfs                     3.0G     0  3.0G   0% /sys/fs/cgroup

Mine has a root filesystem of 12GB, with 3.5GB used and 7.5GB free. If yours has less than a few hundred MB free that is quite likely the cause. If you're unsure, please feel fre to post your output.

If everything is still looking ok for now, you could try manually clearing the caches. I'm not sure that this will help, but perhaps? Do that like this:

rm -r /var/cache/tklbam/*
rm -r /var/cache/duplicity/*

Then retry running 'tklbam-backup'.

If you still get the same issue, then the next thing to check is to re-initialise your server. This will test to see if your current credentials and config are corrupted. We'll keep the current data just in case that doesn't resolve this issue. In an ideal situation, you would have enabled as least "Backup Standard" (@$10/month) so that you can test creation of a new backup (I note that you're currently on "Backup Free). But I'm pretty sure that this process should be able to test re-initialisation without upgrading to a paid plan (even though you won't be able to fully test creation of and uploading) a backup. SO here's what you'll need to do:

mv /var/lib/tklbam /var/lib/tklbam.orig
tklbam-init YOUR_HUB_API_KEY

That should allow you to re-initialise your server as if it was a new server. It should succeed. If it does, you can then move on to test creation of a backup (this step will fail if you don't upgrade to a paid backup plan; but the error should be different to your previous error - it should explicitly note that it can't upload). So again try this:

tklbam-backup

If you get the same error as before and/or the tklbam-init step above gives a similar error message then it seems most likely to me that it must be an issue with your environment (e.g. LAN and/or internet connection). I guess you could test creating a new VM and checking to see if that can connect and if it generates the same errors or not, but I'm not sure how much that will help you diagnose. It would probably require capturing network traffic and comparing to traffic from a working system.

Jeremy Davis's picture

The logs that I have access to aren't particularly detailed, but the only notes regarding your Hub account that I can see are when you set the account up.

Add new comment