systemd sysv init compatibility mode: how it works and troubleshooting when it breaks

systemd sysv init compatibility mode is magical. That is in the sense that it tries to handle compatibility with sysv init scripts while you are distracted looking somewhere else.

When it works it works well, but when things break it makes troubleshooting more difficult. Especially if you don't understand what's going on behind the curtain.

The first thing you need to understand is that this probably doesn't do what you expect:

/etc/init.d/sysv-init-script start

In sysv the script would actually be executed. Under systemd the execution is intercepted by LSB here at this line in the script's execution:

. /lib/lsb/init-functions

This eventually invokes /lib/lsb/init-functions.d/40-systemd which redirects the execution of the script so that it is invoked by systemd.

So instead of executing the script what's actually happening is this:

systemctl start sysv-init-script.service

sysv-init-script.service doesn't actually exist so systemd.service handles that as a special case by sort of pretending it does.

From systemd.service(5):

If a service is requested under a certain name but no unit configuration
file is found, systemd looks for a SysV init script by the same name
(with the .service suffix removed) and dynamically creates a service
unit from that script. This is useful for compatibility with SysV. Note
that this compatibility is quite comprehensive but not 100%. For details
about the incompatibilities, see the Incompatibilities with SysV[1]
document.

Long story short, systemd interprets the standard LSB headers in the sysv init script so it execute the sysv init script in the context of something resembling a real systemd service.

Behind the curtain these wrapper service units are generated by /lib/systemd/system-generators/systemd-sysv-generator and you can find the result at /run/systemd/generator.late/ where the wrapper usually looks something like this:

# Automatically generated by systemd-sysv-generator

[Unit]
SourcePath=/etc/init.d/sysv-init-script
Description=LSB: A sysV init script
After=remote-fs.target systemd-journald-dev-log.socket

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/etc/init.d/sysv-init-script start
ExecStop=/etc/init.d/sysv-init-script stop

The problem is that when things break it adds a layer of indirection that makes debugging the sysv init script harder.

To disable systemd redirection do this:

export _SYSTEMCTL_SKIP_REDIRECT=1

Now executing your /etc/init.d/sysv-init-script directly will work, and so will shell debugging tricks I find really useful such as:

1. set -ex

   This echos everything the script executes to stdout and terminates the script if any command returns an error (I.e., a non-zero exitcode).

2. inserting an interactive /bin/bash shell somewhere to "inspect" the environment. This works kind of like setting a breakpoint.

Note however that systemd is pretty neat once you get past the learning curve. Usually it will make more sense to reimplement your sysv init script as a systemd service unit. You can eliminate hundreds of line of complicated error prone shell script with a handful of lines of systemd configuration directives that work better than the original.