systemd sysv init compatibility mode: how it works and troubleshooting when it breaks

systemd sysv init compatibility mode is magical. That is in the sense that it tries to handle compatibility with sysv init scripts while you are distracted looking somewhere else.

When it works it works well, but when things break it makes troubleshooting more difficult. Especially if you don't understand what's going on behind the curtain.

The first thing you need to understand is that this probably doesn't do what you expect:

/etc/init.d/sysv-init-script start

In sysv the script would actually be executed. Under systemd the execution is intercepted by LSB here at this line in the script's execution:

. /lib/lsb/init-functions

This eventually invokes /lib/lsb/init-functions.d/40-systemd which redirects the execution of the script so that it is invoked by systemd.

So instead of executing the script what's actually happening is this:

systemctl start sysv-init-script.service

sysv-init-script.service doesn't actually exist so systemd.service handles that as a special case by sort of pretending it does.

From systemd.service(5):

If a service is requested under a certain name but no unit configuration
file is found, systemd looks for a SysV init script by the same name
(with the .service suffix removed) and dynamically creates a service
unit from that script. This is useful for compatibility with SysV. Note
that this compatibility is quite comprehensive but not 100%. For details
about the incompatibilities, see the Incompatibilities with SysV[1]
document.

Long story short, systemd interprets the standard LSB headers in the sysv init script so it execute the sysv init script in the context of something resembling a real systemd service.

Behind the curtain these wrapper service units are generated by /lib/systemd/system-generators/systemd-sysv-generator and you can find the result at /run/systemd/generator.late/ where the wrapper usually looks something like this:

# Automatically generated by systemd-sysv-generator

[Unit]
SourcePath=/etc/init.d/sysv-init-script
Description=LSB: A sysV init script
After=remote-fs.target systemd-journald-dev-log.socket

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/etc/init.d/sysv-init-script start
ExecStop=/etc/init.d/sysv-init-script stop

The problem is that when things break it adds a layer of indirection that makes debugging the sysv init script harder.

To disable systemd redirection do this:

export _SYSTEMCTL_SKIP_REDIRECT=1

Now executing your /etc/init.d/sysv-init-script directly will work, and so will shell debugging tricks I find really useful such as:

  1. set -ex

    This echos everything the script executes to stdout and terminates the script if any command returns an error (I.e., a non-zero exitcode).

  2. inserting an interactive /bin/bash shell somewhere to "inspect" the environment. This works kind of like setting a breakpoint.

Note however that systemd is pretty neat once you get past the learning curve. Usually it will make more sense to reimplement your sysv init script as a systemd service unit. You can eliminate hundreds of line of complicated error prone shell script with a handful of lines of systemd configuration directives that work better than the original.

Comments

John Carver's picture

I have a situation where I need to patch a sysv init script and then get systemd to recognize the change.  The change has been applied to /etc/init.d/lxc, but is not being recognized by 'service restart lxc'.  Apparently the change must also be applied to /usr/lib/x86_64-linux-gnu/lxc/lxc-autostart-helper, which appears to be a verbatim copy of the original /etc/init.d/lxc.

I tried copying the patched sysv init script to overwrite the lxc-autostart-helper and then ran 'systemctl daemon-reload'.  This seemed to work okay, but I doubt it is the proper way to accomplish the task.

Information is free, knowledge is acquired, but wisdom is earned.

Janson's picture

We are planning to move httpd service to a Centos7 machine, in my current Centos6 configuration I have a pre-init script which will be executed before starting apache. The script will accept the user sommand line argument(for ex: serice httpd start) and check the following and execute few things before start the service. I am wondering the same can be done in Centos7 with systemd. Any help will be appeciated. 

 

[ "$1" == "start" -o "$1" == "restart" -o "$1" == "condrestart" ]

Jeremy Davis's picture

But re SystemD, it sounds like you will want to use the ExecPreStart= directive in your service file. I would assume that would do what you are hoping for?!

The Systemd service doc page is pretty handy IMO.

Assuming that there is an existing service file you wish to update, the correct way to do that is either copy the file (from /lib/systemd/system/...) to /etc/systemd/system/ and edit the copy in /etc (the file in /etc will be used instead of the one in /lib). Or create a file with a path like /etc/systemd/system/MY_SERVICE.service.d/override.conf and include overrides/additions there. IMO the easiest way to do the latter is use the "systemctl edit MY_SERVICE" command.

Alternatively, it's also an option to call an init.d script from a Systemd service file. E.g. stuff like this:

ExecStart=/etc/init.d/MY_SERVICE start
ExecStop=/etc/init.d/MY_SERVICE stop

Pages

Add new comment