Christoph Knittel's picture

Hi all,

I just discovered that, in the OpenLDAP appliance, the ldap database (located in /var/lib/ldap) is not backed up by tklbam.

I consider this a hazard, as I would definitely expect an OpenLDAP appliance to back up its ldap database by default.

Adding /var/lib/ldap to /etc/tklbam/overrides fixes the problem, but I am not sure whether it is ok to backup the DB while slapd is running.

Christoph

Forum: 
Shak's picture

Looks like there are massive gaps in the Redmine default backup set too. Seems very odd, I can imainge alot of people witha false sense of security.

Is there a decent way to see what has been backed up?

Jeremy Davis's picture

If you use the '-s' switch (for simulate) it will show you everything that will be backed up...

If you have some suggestions on what should be being backed up but isn't please feel free to detail it.

Christoph Knittel's picture

In the meantime I found that it is definitely not ok to backup or restore /var/lib/ldap without shutting down slapd first.

Therefore I added a tklbam hook /etc/tklbam/hooks.d/ldap as follows:

#!/bin/bash

tklbam_operation=$1
tklbam_state=$2

if [ "$tklbam_state" == "pre" ]; then
    service slapd stop
elif [ "$tklbam_state" == "post" ]; then
    service slapd restart
fi
Jeremy Davis's picture

Thanks for posting this. Sounds like it should be included in the TKLBAM OpenLDAP profile.

Also I have just raised a bug on the TKL Isue Tracker.

Howard Chu's picture

You can take an LDIF backup of back-bdb, back-hdb, and back-mdb while slapd is running, using slapcat. Using standard filesystem tools (like tar, etc.) definitely won't work. This is all documented in the OpenLDAP Admin Guide and slapcat(8) manpage. If you want a raw DB dump you need to use the db-specific tools for the job, e.g. BerkeleyDB's db_hotbackup or LMDB's mdb_copy.

 

Christoph Knittel's picture

Howard, thanks for the hint! Unfortunately I am not an expert on OpenLDAP or BerkeleyDB.

In the OpenLDAP admin guide, it says

19.4. Migration

The simplest steps needed to migrate between versions or upgrade, depending on your deployment type are:

  1. Stop the current server when convenient 
  2. slapcat the current data out 
  3. Clear out the current data directory (/usr/local/var/openldap-data/) leaving DB_CONFIG in place 
  4. Perform the software upgrades
  5. slapadd the exported data back into the directory
  6. Start the server

So I guess

  • this procedure could be used in the tklbam pre/post hooks and
  • this would the safest way to go (especially in the case of restoring to a new instance of the OpenLDAP appliance that has a newer version of slapd).

Do you have any sample backup/restore scripts based on slapcat/slapadd and covering all the dbs?

Howard Chu's picture

No, we have no sample scripts. I don't think anyone has ever felt it was complicated enough to need a dedicated script.

Eric's picture

The LDAP schema's and index definitions are normally restored because they are stored in "/etc/ldap/slapd.d/" (and below).

To backup the LDAP data fields I use the (modified example) script "/etc/tklbam/hooks.d/ldap" (Note: /root/ldap/backup.ldif wil be normally backuped and restored):

#!/bin/bash -e
op=$1
state=$2
if [ "$state" = "pre" ]; then
    service slapd stop
elif [ "$state" = "inspect" ]; then
    if [ "$op" = "restore" ]; then
    elif [ "$op" = "backup" ]; then
        slapcat -l /root/ldap/backup.ldif
    fi
elif [ "$state" = "post" ]; then
    service slapd restart
else
    echo "bad hook invocation"
fi

I manually restore the LDAP data fields because a clean appliance doesn't have the ldap hook script with the following script:

#!/bin/bash
service slapd stop
rm -f /var/lib/ldap/*
slapadd -F /etc/ldap/slapd.d/ -l /root/ldap/backup.ldif
chown openldap:openldap /var/lib/ldap/*
service slapd start

Regards

Howard Chu's picture

You should use slapcat -n0 to backup the config. The slapd.d files are an slapd internal database and database file formats are subject to change without notice. (And slapadd -n0 to restore from LDIF.) slapcat/slapadd LDIF is the only officially supported mechanism for backup/restore, and this is already clearly stated in all of the documentation.

Eric's picture

Thanks to point this out, I have changed the scripts as follows and it works great.

Hook:

#!/bin/bash -e
op=$1
state=$2
if [ "$state" = "pre" ]; then
    service slapd stop
elif [ "$state" = "inspect" ]; then
    if [ "$op" = "restore" ]; then
        echo "Only manual restore"
    elif [ "$op" = "backup" ]; then
        slapcat -n0 -l /root/ldap/config.ldif
        slapcat -l /root/ldap/data.ldif
    fi
elif [ "$state" = "post" ]; then
    service slapd restart
else
    echo "bad hook invocation"
fi

Restore

#!/bin/bash
service slapd stop
# config
rm -rf /etc/ldap/slapd.d/*
slapadd -n0 -F /etc/ldap/slapd.d/ -l /root/ldap/config.ldif
chown -R openldap:openldap /etc/ldap/slapd.d/*
# data
rm -rf /var/lib/ldap/*
slapadd -F /etc/ldap/slapd.d/ -l /root/ldap/data.ldif
chown -R openldap:openldap /var/lib/ldap/*
service slapd start

 

Jeremy Davis's picture

I have added additional notes to the bug report referring back to this thread.

Christoph Knittel's picture

Hi guys,

Thanks a lot for your work! I would have looked into this myself after Howard's comments, but didn't find the time. I will test the latest version of Eric's scripts and let you know my findings.

Christoph

Eric's picture

Using my scripts I discover that the first time I make a backup the config.ldif and data.ldif files doesn't appear in the backup! In following backup's they are in the backup.

This means, to my opinion, that the check of files to be backuped is done before calling the hook scripts. I don't know where to change the order.

Christoph Knittel's picture

I can confirm the issue that Eric mentions above. It can be resolved by performing the backup in the operation=backup / state=pre phase.

Also, in Eric's script, an error would occur if the backup directory didn't exist yet.

Here is my latest version of the script with backup and restore working perfectly for me: :-)

#!/bin/bash

tklbam_operation=$1
tklbam_state=$2

ldap_backup_dir=/root/ldap_backup

if [ "$tklbam_operation" == "backup" ] && [ "$tklbam_state" == "pre" ]; then
    service slapd stop

    mkdir -p $ldap_backup_dir
    slapcat -n0 -l $ldap_backup_dir/config.ldif
    slapcat -l $ldap_backup_dir/data.ldif

    service slapd start
fi

if [ "$tklbam_operation" == "restore" ] && [ "$tklbam_state" == "post" ]; then
    service slapd stop

    # restore config
    rm -rf /etc/ldap/slapd.d/*
    slapadd -n0 -F /etc/ldap/slapd.d/ -l $ldap_backup_dir/config.ldif
    chown -R openldap:openldap /etc/ldap/slapd.d/*

    # restore data
    rm -rf /var/lib/ldap/*
    slapadd -F /etc/ldap/slapd.d/ -l $ldap_backup_dir/data.ldif
    chown -R openldap:openldap /var/lib/ldap/*

    service slapd start
fi
Eric's picture

I can confirm that the script above works, thanks Christoph.

I have noticed errors from the "libnss-ldapd" package during backup, so I stop/start also the "nslcd" service.

Another point I found is that during "--simulate" of tklbam-backup and tklbam-restore the script is normally executed :-( Is it possible that the script is aware of the "--simulate" option?

Liraz Siri's picture

Thanks to everyone who helped develop and test this hook. Awesome work guys.

I monkey patched the TKLBAM profiles for OpenLDAP that live on the Hub to include this hook so that that slapd migration happens out of the box and you don't need to install hooks to /etc/tklbam/hooks.d

It's been possible to do this since TKLBAM v1.4. From the RelNotes-1.4:

* Cryptographically signed profile-level hooks are now supported. It's a
  generic mechanism but the original motivation for this was to allow us
  to automatically fix migrations between two versions of an appliance
  (E.g., TurnKey Redmine 12 and TurnKey Redmine 13)
Redmine was the first and only use case up until now. Thanks for helping me add a second one!

So that I don't have to monkey patch tklbam profiles for future versions I added the final version of the hook by Christoph to tklbam-profiles:

https://github.com/turnkeylinux/tklbam-profiles/commit/d12462fa18d2f721a...

Jeremy Davis's picture

I just closed the issue on the tracker.

Christoph Knittel's picture

Thanks a lot!

I just retested backup and restore to a new cloud server without my script present in /etc/tklbam/hooks.d, and everything worked fine. :-)

Jeremy Davis's picture

Great news! Thanks for the confirmation. :)

Add new comment