Christian's picture

Hi, I have seen that it is possible to use TKLBAM with any Debian-based installation. But what will be backed up once TKLBAM is initialized? Only the MySQL database and any files added/changed/deleted since that point in time? Or also all files that are not controlled by the package management, i.e. that have been added manually? I didn't fully understand the logic.

Thanks for any information.

Forum: 
Jeremy Davis's picture

I have it installed on my Debian laptop but I don't recall the exact process I went through to set it up, it was a while ago now... I really should have documented it!

If you are installing on vanilla Debian then you need to create your own custom profile. All the existing TurnKey appliances have their own profile which tells TKLBAM which directories (and databases as relevant) to include and/or exclude...

My suggestion would be to install it; set yourself up a basic blank profile and have a play. The docs are pretty good; although obviously geared towards existing appliances so you'll need to read between the lines sometimes... Staret with tklbam-backup and test it out. You can use the "-s" swith to simulate and/or the "--dump=/path/to/dir" to dump your backup to a directory before you start actually uploading stuff!

Let us know how you go. If you have any issues I'll try to help you out. Perhaps if you end up documenting your process you could post it here!?

Christian's picture

Thanks Jeremy for the quick answer. Ok, I did not know that it works with manually configured profiles - I thought that TKLBAM would somehow infer the directories from the package management metadata. Too bad.

What I want to do is to back up the content of a docker container based on a vanilla debian, with plugins added. So there are three states: 1) the vanilla debian, 2) the default initial state after the dockerfile has run and installed all dependencies, and 3) finally the current state of the application, with files and database content. Ideally, I can produce state 2 in a new docker container and then restore state 3 from a TKLBAM backup.

If I understand correctly, I need to create a profile for state 2, then.

 

 

 

Christian's picture

# wget -O - -q https://raw.github.com/turnkeylinux/tklbam/master/contrib/ez-apt-install.sh | PACKAGE=tklbam /bin/bash 
# apt-get install tklbam
# tkbam init MYSECRETAPIKEY

Then it will tell you that profile "Debian 7" (yay!!!) has been downloaded and the a link to the Hub has been established.

Doing a dry run revealed that all additions to the base Debian are saved.

Great stuff. I'll do a test restore in a different container to see if it works.

 

Christian's picture

I created a clone of the current container, installed tklbam there and tried a restore. this is what happened:

 

root@container2:~# tklbam restore 25
Executing Duplicity to download s3://s3-eu-west-1.amazonaws.com/tklbam-adsafdsaas to /tmp/tklbam-b4PY04
==============================================================================================================

// started squid: caching downloaded backup archives to /var/cache/tklbam/restore

Traceback (most recent call last):
  File "/usr/bin/tklbam", line 46, in <module>
    CliWrapper.main()
  File "/usr/lib/tklbam/cliwrapper.py", line 88, in main
    commands[command].main()
  File "/usr/lib/tklbam/cmd_restore.py", line 491, in main
    backup_extract_path = get_backup_extract()
  File "/usr/lib/tklbam/cmd_restore.py", line 463, in get_backup_extract
    downloader(raw_download_path, target, log=_print if not silent else None, debug=opt_debug, force=opt_force)
  File "/usr/lib/tklbam/duplicity.py", line 166, in __call__
    squid.start()
  File "/usr/lib/tklbam/squid.py", line 59, in start
    raise Error("%s failed to start\n" % SQUID_BIN + self.command.output)
squid.Error: /usr/lib/tklbam/deps/usr/sbin/tklbam-squid failed to start
FATAL: storeUfsDirParse: invalid size value
Squid Cache (Version 2.7.STABLE9): Terminated abnormally.
CPU Usage: 0.003 seconds = 0.002 user + 0.001 sys
Maximum Resident Size: 5536 KB
Page faults with physical i/o: 0
Aborted
Traceback (most recent call last):
  File "/usr/lib/tklbam/deps/usr/sbin/tklbam-squid", line 192, in <module>
    main()
  File "/usr/lib/tklbam/deps/usr/sbin/tklbam-squid", line 188, in main
    system("%s -f %s -z" % (squid_bin, conf_tmp.path))
  File "/usr/lib/python2.7/dist-packages/executil.py", line 56, in system
    raise ExecError(command, exitcode)
executil.ExecError: non-zero exitcode (134) for command: /usr/lib/tklbam/deps/usr/sbin/squid -f /tmp/tklbam-squid-conf-gWw287 -z
Jeremy Davis's picture

Looks like something has gone wrong with squid. Perhaps try running/testing/checking logs with/for that?
Christian's picture

Maybe Alon or Liraz could have a look, since they wrote TKLBAM, they should know right away what might be the problem?

Jeremy Davis's picture

Maybe they might know but I'd rather not interrupt them until you've done the simple troubleshooting steps first...
Christian's picture

Ok. Where do I find the logfiles? There is /var/log/tklbam-restore, but it contains the same error backtrace as was dumped to stdout. There is no special log file for squid. Where could I look?

 

Jeremy Davis's picture

I just had a look on my Wheezy laptop (where I have it installed and working) and I can't see any specific log files for tklbam-squid (or anything tklbam much). TBH I can't find much about it at all... :(

But looking over your stacktrace again I just tried to launch squid-cache to get a feel for what might be going wrong.

$ /usr/lib/tklbam/deps/usr/sbin/tklbam-squid
error: incorrect number of arguments
Syntax: /usr/lib/tklbam/deps/usr/sbin/tklbam-squid  
Runs a squid instance to cache TKLBAM's duplicity archives
Arguments:

             [hostname:]port (e.g., 3128, 127.0.0.1:33128)
             Size of squid cache (can be absolute or relative size)
        
                        1000MB - 1000 MBs
                        2GB - 2000 MBs

                        50%                                                  of free space in spool location

Environment variables:

    TKLBAM_SQUID_CACHE_DIR      location of cache dir (default: /var/spool/tklbam-squid)
    TKLBAM_SQUID_USER           user under which we execute squid (default: proxy)
    TKLBAM_SQUID_BIN            path to tklbam squid binary (default: /usr/lib/tklbam/deps/usr/sbin/squid)

Do you have plenty of free space? I wonder if that might be the issue? Although I'm only guessing really...

Regardless I'll email Alon & Liraz. TKLBAM is actually Liraz's baby so I'm not sure if Alon will know. Liraz will but he is in "dev mode" and is notoriously hard to get hold of... I'll let you know...

Christian's picture

Hi Jeremy, any news from Liraz?

Christian's picture

Thank you Jeremy, much appreciated. TKLBAM would be a critical component of the backup strategy of a dockerized application which cannot be based on an TKL appliance (since there are no dockerized TKL-Appliances yet). 

Christian's picture

Alternatively, if trying to add TKLBAM to an existing dockerized image doesn't work out, I might as well try to rewrite my Dockerfile to be based on the TKL-Core Docker image. Let's see whether Liraz finds time to look at the problem in the next couple of days.

Christian's picture

Ok, now I have replaced the original non-TKL debian install with a TKL based docker image, only to end up with the same problem that has something to do with squid. See the attached screenshot (I cannot copy & paste from the web-based terminal).

I hope that Liraz can have a look, because this getting to be a REAL problem.

Thank you!

 

Jeremy Davis's picture

In retrospect I should have added this issue to the tracker. FWIW I have just done that: https://github.com/turnkeylinux/tracker/issues/502

I have also sent Liraz a message so hopefully he will get back to me/us real soon...

Christian's picture

Thank you. Maybe it is a Docker-specific bug and has nothing to do with the distribution itself? The problem occurred in both cases in Docker containers. I have added this in the bugtracker.

Add new comment