TurnKey Linux Virtual Appliance Library

Design overview

Read "we set out to design the ideal backup system" for a less formal overview.

Key elements

TurnKey Hub: a web service which provides the front-end for backup management. The user links an appliance to a specific Hub account identified by an API KEY.

Webmin module: extends Webmin to provide a convenient alternative to command line usage that you can use from any web browser. Log into your appliance's Webmin interface to see it in action.

Backup profile: describes the installation state for a specific type and version of appliance. An appropriate profile is downloaded from the Hub the first time you backup, or as required if there is a profile update (e.g., bugfix).

Delta: a set of changes since installation to files, users, databases and package management state. This is calculated at backup time by comparing the current system state to the installation state described by the backup profile.

Encryption key: generated locally on your server and used to directly encrypt your backup volumes. By default key management is handled transparently by the Hub. For extra security, the encryption key may be passphrase protected cryptographically. An escrow key can be created to protect against data loss in case the password is forgotten.

Duplicity: back-end primitive that the backup and restore operations invoke to encode, transfer and decode encrypted backup volumes which contain the delta. It communicates directly with the storage target (e.g., Amazon S3). In normal usage the storage target is auto-configured by the Hub. Duplicity uses the rsync algorithm to support efficient incremental backups. It uses GnuPG for symmetric encryption (AES).

Amazon S3: a highly-durable cloud storage service where encrypted backup volumes are uploaded to by default. To improve network performance, backups are routed to the closest datacenter, based on a GeoIP lookup table.

Any storage target supported by Duplicity can be forced but this complicates usage as the Hub can only work with S3. This means backups, encryption keys and authentication credentials will need to be managed by hand.

Principle of operation

Every TKLBAM-supported TurnKey appliance has a corresponding backup profile that describes installation state and includes an appliance-specific list of files and directories to check for changes. This list does not include any files or directories maintained by the package management system.

A delta (I.e., changeset) is calculated by comparing the current system state to the installation state. Only this delta is backed up and only this delta is re-applied on restore.

An exception is made with regards to database contents. These are backed up and restored whole, unless otherwise configured by the user.

In addition to direct filesystem changes to user writeable directories (e.g., /etc, /var/www, /home) the backup delta is calculated to include a list of any new packages not originally in the appliance's installation manifest. During restore, the package management system is leveraged to install these new packages from the configured software repositories.

Users and groups from the backed up system are merged on restore. If necessary, uids / gids of restored files and directories are remapped to maintain correct ownership.

Similarly, permissions for files and directories are adjusted as necessary to match permissions on the backed up system.