David Johnson's picture

I have a couple of OpenVZ VM's webservers built on turnkey 13-0 core which use daily backup to the turnkey hub. On a couple of occasions recently I have had an odd problem. The tklbam app seems to have hung with high cpu load in the final stages of cleaning up the bakup run. The backup appears to have been completed and is present on the hub as expected. The log file show the run completing without error (Error 0) but this is followed by a python trace dump as follows:

Errors 0

Traceback (most recent call last):
  File "/usr/bin/tklbam-backup", line 513, in <module>
  File "/usr/bin/tklbam-backup", line 485, in main
  File "/usr/bin/tklbam-backup", line 417, in backup_inprogress
    hb.set_backup_inprogress(registry.hbr.backup_id, bool)
  File "/usr/lib/tklbam/hub.py", line 232, in set_backup_inprogress
    {'bool': bool})
  File "/usr/lib/tklbam/hub.py", line 179, in _api
    return self.api.request(method, self.API_URL + uri, attrs, headers)
  File "/usr/lib/tklbam/hub.py", line 117, in request
    return _API.request(self, method, url, attrs, headers)
  File "/usr/lib/python2.6/dist-packages/pycurl_wrapper.py", line 193, in request
    return json.loads(response.data)
  File "/usr/lib/pymodules/python2.6/simplejson/__init__.py", line 384, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/pymodules/python2.6/simplejson/decoder.py", line 402, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/pymodules/python2.6/simplejson/decoder.py", line 420, in raw_decode
    raise JSONDecodeError("No JSON object could be decoded", s, idx)
simplejson.decoder.JSONDecodeError: No JSON object could be decoded: line 1 column 0 (char 0)

The hung backup process had to be killed manually at which point cron sends out an email indicating the process failed. On the first occasion it was a couple of days before I was able to attend to it, the following days backup runs completed okay with the original hung process still running.

Any clues?

Thanks, David Johnson

Jeremy Davis's picture

TBH I have no idea, but I'll alert Liraz (TKLBAM Daddy) and see if he has any idea of what might be going on...

Add new comment