TKLBAM fails with gdocs backend

Basically, TKLBAM fails with the gdocs backend but I've almost managed to get it working by installing new versions of duplicity and python-gdata. But not quite... read below for where where I'm at. Please note that there might be some discrepancies in my description as my terminal scroll buffer filled up and as I've tried quite a few things and am going on memory I had to retry things which may have yielded different results to the first attempt given all that went on before!

So... in the beginning I downloaded my profile for my turnkey-wordpress-13.0-wheezy-amd64 virtual appliance (running in VirtualBox) from the HUB using

tklbam-init API-KEY

Backed up the downloaded profile from the registry, and copied the backup for possible subsequent modification:

cp -pR /var/lib/tklbam/profile /etc/tklbam/hub-profile

Then created a custom profile using the hub-profile dirindex.conf:

tklbam-internal create-profile /etc/tklbam/nonhub-profile /etc/tklbam/hub-profile/dirindex.conf

Then disconnected from the HUB (--solo) and loaded my custom profile (--force-profile) just created into the registry (/var/lib/tklbam):

tklbam-init --solo --force-profile=/etc/tklbam/nonhub-profile

So far so good, but then when I go to make a backup:

tklbam-backup --address gdocs://USER:PASS@gmail.com/path/to/backup

Got this error

BackendException: Google Docs backend requires Google Data APIs Python Client Library (see http://code.google.com/p/gdata-python-client/).

and realised I needed python-gdata from the eror message and reading this so that Duplicity can support the gdocs storage backend. Found it already package for Debian so installed it:

apt-get update && apt-get install python-gdata

That fixed this error but then got a new error "'DocsClient' object has no attribute 'get_everything'":

# export PASSPHRASE=$(cat /var/lib/tklbam/secret)
# duplicity --verbosity=5 --archive-dir=/var/cache/duplicity cleanup --force gdocs://USER:PASS@gmail.com/path/to/backup

Backend error detail: Traceback (most recent call last):
[...]
  File "/usr/lib/tklbam/deps/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py", line 182, in __handle_error
    raise BackendException(message)
BackendException: Error while fetching remote entries: 'DocsClient' object has no attribute 'get_everything'.

After some googling found someone who had a similar error so I thought that I needed a newer version of python-gdata. I tried these versions of the package

python-gdata_2.0.18+dfsg-1_all.deb
python-gdata_2.0.18+dfsg1-2_all.deb

but neither of them fixed the problem.

So after further research here found out that duplicity probably also needed updating. Found out that wheezy-backports had this new version of duplicity backported so added

deb http://ftp.uk.debian.org/debian wheezy-backports main

to my apt sources (/etc/apt/sources.list.d/sources.list) and then installed it (forcing the release to the wheezy-backports release with the "-t" option otherwise it won't be installed automatically):

apt-get update && apt-get install -t wheezy-backports duplicity

This still didn't fix the error but I then realised that tklbam runs its own version of duplicity!

/usr/lib/tklbam/deps/bin/duplicity

rather than the vanilla one

/usr/bin/duplicity

So backed up /usr/lib/tklbam/deps/bin/duplicity:

mv /usr/lib/tklbam/deps/bin/duplicity /usr/lib/tklbam/deps/bin/duplicity.v.0.6.18

and replaced it with a script which calls through to the backported duplicity (version 0.6.24-1~bpo70+1 in /usr/bin/duplicity) by creating a shelll script called duplicity in this directory (/usr/lib/tklbam/deps/bin) and making it executable:

cat >| /usr/lib/tklbam/deps/bin/duplicity <<'END'
#!/bin/sh
export PYTHONPATH=""
/usr/bin/duplicity $@
END
chmod +x /usr/lib/tklbam/deps/bin/duplicity

And voila tklbam-backup now worked!

Although, initially I did have a different error:

ImportError: cannot import name progress

since I didn't set

export PYTHONPATH=""

which meant that python wasn't using the vanilla libraries but probalby those in /usr/lib/tklbam/deps/lib/python2.7.

In the longer term could a new version of tklbam be released either including or using the vanilla backported duplicity?

Thanks in advance!

Replies (6)

Interesting adventures! :)

Reply 1 2014-12-09

Sounds like you are nearly there... Unfortunately I can't offer much assistance as I am not familiar with the inner workings of TKLBAM (besides I'm not much of a programmer anyway...). You may already be aware, but TKLBAM's source code is here on GitHub. Also I have added a feature request to the issue tracker.

Adventure over!

Reply 2 2014-12-20

Thanks Jeremy for raising the feature request - would be nice if the gdocs backend worked out of the box so to speak!

After taking a bit of a break from this managed to finally get it working - at least the backup - will test a restore shortly! With some more research found someone with the same problem and it was suggested to reset the PYTHONPATH. Without looking at the tklbam code it seems pretty obvious that it would be setting PYTHONPATH to /usr/lib/tklbam/deps/lib/python2.7 before calling duplicity to ensure the correct Python modules for this particular version of duplicity get loaded.

For those interested I have updated my work-around documented in my initial post with the solution I found which worked - for both backup and restores.

backports is a special case

Reply 3 2014-12-22

Under normal circumstance any repo you add to sources.list (FWIW actually I personally prefer to add a new '.list' file to sources.list.d to keep them all separate) will automatically be considered as a valid source for 'apt-get install <package-name>' and 'apt-get upgrade'. So under normal circumstance you need to be really careful when adding repos that have duplicate packages (with different versions). It can often be a fast track to dependency hell! The only safe way to do it really is to pin packages.

However in their wisdom the Debian devs decided that backports is likely to be added by newbs and so pre-configurd it (if/when it's added) to be user friendly.

The default config is that the backports repo is ignored for 'apt-get install' - unless of course it is specified by '-t wheezy-backports' (followed by the package names you want from backports - package names preceeding the '-t' will come from normal repos). IIRC using the '-t backports' will also automatically pull any dependencies (if available) from backports too, which is probably what you'd want. 'apt-get upgrade' will also ignore packages in backports, unless they were installed from backports - then they will be upgraded from backports... So yes you are right, if a newer version of duplicity gets in there is will be updated. I doubt that will be an issue though.

Thanks for the tip

Reply 4 2014-12-20

You're for that tip Jeremy - you're very helpful as usual! I'm still getting up to speed with apt-get and wasn't 100% under which cirucmstances "apt-get upgrade" would use packages from this wheezy-backports release and to stay on the safe side I advised to comment out this release. But it's good to know I can leave it in my sources knowing that it will only be used if I specify "-t wheezy-backports".

I have also updated my original post with your suggestion as it is is a bit simpler, although in the future could potentially install a different and newer backported version of duplicity although hopefully that would work as well!

Thanks for posting back

Reply 5 2014-12-16

And IIRC the only reason why the core devs created a 'turnkey' duplicity fork was because there was some bugfixes that were in a more recent branch than the one that came with Wheezy. Following your discoveries, it sounds like they may be able to rebase back on the default duplicity when TKL rebases in Jessie (for v14).

Also it shouldn't be an issue re having the backport repo enabled, as by default you should have to specify installing from that. On my desktop (Wheezy) system I have backports enabled and to explicitly install from backports you need to do:

apt-get install -t wheezy-backport <some-package-from-backports>

Adventure completely over!

Reply 6 2014-12-13

I tested tklbam-restore and it also worked fine with the /usr/bin/duplicity I hacked tklbam to use.

I also re-tested the above hacks on a fresh install of TKL Wordpress and I found it not necessary to upgrade python-gdata beyond the version available in the Debian wheezy release (version 2.0.17+dfsg-1).

This means that you only need to run

apt-get install python-gdata

to get a version of python-data that will work with the gdocs backend as of today with the backported duplicity 0.6.24-1~bpo70+1 from the Debian wheezy backports release. That is don't download any python-gdata packages nor install them with "dpkg -i".

It's probably also a good idea to remove the line or comment out (precede with "#")

deb http://ftp.uk.debian.org/debian wheezy-backports main

from your apt sources (/etc/apt/sources.list.d/sources.list) after installing the backported duplicity to avoid unnecessary upgrades when running "apt-get upgrade".