You are here
Jack Riddle - Thu, 2020/06/18 - 21:15
Backup ran find for a few months, but now when I try to run tklbam-backup, I get the following error:
warning: using cached profile because of a Hub error: error(77, '') error: error(77, '')
I work with linux out of necessity and infrequently, so troubleshooting this is not easy for me. Anyone have some guidance on what I should try?
Forum:
Hi Jack, doesn't sound good!
Ok, so the error suggests that the issue is somehow related to your servers connection to the Hub. Perhaps there is something wrong with your Hub account? Or perhaps something has gone wrong on your server? If your server is running locally, then perhaps the issue lays somewhere in between (e.g. your internet connection or local network)?
So I would suggest that you try logging into your Hub account and see if there is anything obvious there. Please feel free to ping me there via the "Support" link towards the left in the top bar, or the blue chat icon in the bottom right corner. I can then have a look from our end too and see if there is anything that jumps out.
To troubleshoot your server, can you please give me the output of the following commands:
If you server is running locally, then I'd suggest that you investigate any local networking changes. E.g. a new ISP or changed internet plan; a new or changed proxy server to connect to the internet; a new or changed firewall setup; etc.
Small change, but should have no effect.
I don't see any issues in the hub. There have been no changes to our firewall or internal environment, but our bandwidth was recently upgraded. However, this was handled by our ISP on their own equipment--no changes were made to our environment.
I can see the turnkey server hit our firewall when I try to run TKLBAM-Backup, and the firewall indicates the traffic is allowed:
Here are the results from the requested commands:
Ok, so you have the latest TKLBAM release for v15.x.
Considering that you have the latest TKLBAM, everything looks ok in your Hub account, ping is responding as expected, and that I can't reproduce this issue in a local VM from my end (here in Australia), it is almost certainly something within your VM and/or your environment (e.g. host system if running as a VM; networking within your LAN and/or infrastructure including internet connection). To break it down a bit further, my guess is one of:
So unless connecting to the internet via an alternate path is an easy option and/or you want to contact your ISP and explicitly ask them about any changes/difference that may affect encrypted traffic, let's put the ISP possibility aside. (Also FWIW even if you do contact your ISP they may deny any changes regardless).
So to focus on potential issues related to your local TurnKey appliance, assuming that it's a VM, please double check the host system has plenty of free space. If your VM has a dynamically assigned disk, if the host runs out of free space (or is very close), then the guest may act as if it has also run out of space despite showing that it still has free space. I'd also suggest checking system logs on the host to
Assuming that's ok, within your TurnKey appliance, check the available disk space like this:
Look for the line that is "Mounted on" "/" (it should either be '/dev/mapper/turnkey-root" or "/dev/sda1"). Then double check how much is "Avail" (available). E.g. here's the output of a VM I have handy:
Mine has a root filesystem of 12GB, with 3.5GB used and 7.5GB free. If yours has less than a few hundred MB free that is quite likely the cause. If you're unsure, please feel fre to post your output.
If everything is still looking ok for now, you could try manually clearing the caches. I'm not sure that this will help, but perhaps? Do that like this:
Then retry running 'tklbam-backup'.
If you still get the same issue, then the next thing to check is to re-initialise your server. This will test to see if your current credentials and config are corrupted. We'll keep the current data just in case that doesn't resolve this issue. In an ideal situation, you would have enabled as least "Backup Standard" (@$10/month) so that you can test creation of a new backup (I note that you're currently on "Backup Free). But I'm pretty sure that this process should be able to test re-initialisation without upgrading to a paid plan (even though you won't be able to fully test creation of and uploading) a backup. SO here's what you'll need to do:
That should allow you to re-initialise your server as if it was a new server. It should succeed. If it does, you can then move on to test creation of a backup (this step will fail if you don't upgrade to a paid backup plan; but the error should be different to your previous error - it should explicitly note that it can't upload). So again try this:
If you get the same error as before and/or the tklbam-init step above gives a similar error message then it seems most likely to me that it must be an issue with your environment (e.g. LAN and/or internet connection). I guess you could test creating a new VM and checking to see if that can connect and if it generates the same errors or not, but I'm not sure how much that will help you diagnose. It would probably require capturing network traffic and comparing to traffic from a working system.
FWIW, I just double checked the Hub logs and no errors there
The logs that I have access to aren't particularly detailed, but the only notes regarding your Hub account that I can see are when you set the account up.
Working from home slowed me down
I had other priorities as school started back that limited my ability to work on this further. Now, I'm back at it.
First, I checked space. All my values equal or exceed your own example. The host has plenty fo free space.
Second, I cloned the VM and connected it directly to a separate internet connect--a cable modem in bridge mode--so there wasn't a firewall or anything that could interfere. No change.
Third, I attempted the re-initialise steps. After running tklbam-init with my hub key, I get the "Generated backup encryption key" message, and then "error: error(77, ' ')"
Running the command again returns the same error with the encryption key message.
t
My ultimate plan for getting this to work again is to migrate to a new Turnkey appliance. If there is another easy way to do that other than TKLBAM, I can put this to bed and do it another way since this server will go bye bye.
This is such a bizarre issue?!
I don't understand why this is occurring?! It sounds like you have pretty much eliminated all the possibilities that I can think of?!
I've dug into the TKLBAM code a bit and given the error message that you have reported, it still appears to be related to communication with the Hub!? FWIW, line 185 is the only place in tklbam-init where it can just return a simple "error: ERROR_MESSAGE" (without any further info).
Following that thread, hub.Backups.get_sub_apikey(apikey) seems to be what's rasing the error. Looking closer at that though, it appears to be a fairly straight forward call the the Hub API. And the error that is being returned, is not one that is expected or accounted for. The "expected" possible errors all appear to be clearly documented. So I think that we can assume that it's not one of them
So my guess at this point is that either there is (still) something between you and the Hub that is mangling the traffic somewhere/somehow. Or there is something broken in your Hub account, that is not giving us a clear and/or expected error message.
So at this point, I see 2 paths forward:
Option 1 - hack on TKLBAM
This won't actually fix the issue (and may still lead to a dead end). But it will hopefully give me more insight into what is actually wrong. You can see the changes that I've made here. And here is the command to run to download it and overwrite the default file:
Then rerun the previous command and post the output.
Option 2 - Continue "brute force" approach
The next step in this approach is to just reinstall TKLBAM and cross your fingers:
It may also be worth reinstalling it's dependencies, but the only one that should make a difference at this point is 'pycurl-wrapper':
Option 3 - Collect and migrate data via some other method
Essentially all you need is a copy of the 'wordpress' database and the relevant files (should all be in /var/www/wordpress).
If you have some idea of what you're doing with WrodPress and Linux, then you could just manually do that yourself, zip it all up and copy it across to a new server with SFTP or Rsync.
If you aren't so confident, then I suggest that either you do a bit of googling on how to do that (you should find plenty of tutorials). TurnKey is based on Debian and this is something that is generic enough that even tutorials based on other Linux distros would probably "just work" too.
Another option is to look for a backup WordPress plugin. That will take care of the heavy lifting for you and you can probably just install it on the new TurnKey server too and restore your backup using the same plugin.
Regardless of which path you take. Please let me know how you go and don't hesitate to ask if you have further questions/concerns/errors and I'll do my best to help out.
Going to try all in case it happens again to me or another.
For option 1, I did as requested and got the following output:
For option 2, updated apt-get and ran the reinstall. Same error message.
I'll persue option 3 in the meantime.
Ok, so some more troubleshooting...
Thanks for that. After looking more closely at the code, I actually think that the root issue is outside TKLBAM; perhaps something not quite right (perhaps even an undocumented/missing dependency?) in python-pycurl or pycurl-wrapper.
To continue down the Option 1 path, and extend the stacktrace (and perhaps confirm my suspicions) I have another file to update temporarily for troubleshooting:
Again if you wish to checkout what I changed you can see it here. FWIW, similar to last time, I commented out error handling code to allow more verbose stacktrace.
IMPORTANT - Please be sure to double check that your Hub API key is not included in the output that you post. FWIW it shouldn't this time, but if we continue I'll possibly include some "print" statements so we can see the data that is being passed (to ensure that it's valid). When that happens there is risk that your API key will be in the output. If it is, please replace it with a string that make it clear, e.g. MY_HUB_API_KEY or similar.
I was also think that it might be worth reinstalling some other packages (i.e. a continuation of "Option 2"), but I'm not sure of the value there, or even exactly which packages are more likely to be causing the issue, so I'll just leave that be for now...
It is all Greek to me.
Here is the output with the 2nd file applied. (First time I ran it, there was no output. I assume this is due to the tklbam reinstall. I reapplied the first file and then the second, and ran the init command and got this output.)
Add new comment