systemd sysv init compatibility mode: how it works and troubleshooting when it breaks

systemd sysv init compatibility mode is magical. That is in the sense that it tries to handle compatibility with sysv init scripts while you are distracted looking somewhere else.

When it works it works well, but when things break it makes troubleshooting more difficult. Especially if you don't understand what's going on behind the curtain.

The first thing you need to understand is that this probably doesn't do what you expect:

/etc/init.d/sysv-init-script start

In sysv the script would actually be executed. Under systemd the execution is intercepted by LSB here at this line in the script's execution:

. /lib/lsb/init-functions

This eventually invokes /lib/lsb/init-functions.d/40-systemd which redirects the execution of the script so that it is invoked by systemd.

So instead of executing the script what's actually happening is this:

systemctl start sysv-init-script.service

sysv-init-script.service doesn't actually exist so systemd.service handles that as a special case by sort of pretending it does.

From systemd.service(5):

If a service is requested under a certain name but no unit configuration
file is found, systemd looks for a SysV init script by the same name
(with the .service suffix removed) and dynamically creates a service
unit from that script. This is useful for compatibility with SysV. Note
that this compatibility is quite comprehensive but not 100%. For details
about the incompatibilities, see the Incompatibilities with SysV[1]
document.

Long story short, systemd interprets the standard LSB headers in the sysv init script so it execute the sysv init script in the context of something resembling a real systemd service.

Behind the curtain these wrapper service units are generated by /lib/systemd/system-generators/systemd-sysv-generator and you can find the result at /run/systemd/generator.late/ where the wrapper usually looks something like this:

# Automatically generated by systemd-sysv-generator

[Unit]
SourcePath=/etc/init.d/sysv-init-script
Description=LSB: A sysV init script
After=remote-fs.target systemd-journald-dev-log.socket

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
ExecStart=/etc/init.d/sysv-init-script start
ExecStop=/etc/init.d/sysv-init-script stop

The problem is that when things break it adds a layer of indirection that makes debugging the sysv init script harder.

To disable systemd redirection do this:

export _SYSTEMCTL_SKIP_REDIRECT=1

Now executing your /etc/init.d/sysv-init-script directly will work, and so will shell debugging tricks I find really useful such as:

  1. set -ex

    This echos everything the script executes to stdout and terminates the script if any command returns an error (I.e., a non-zero exitcode).

  2. inserting an interactive /bin/bash shell somewhere to "inspect" the environment. This works kind of like setting a breakpoint.

Note however that systemd is pretty neat once you get past the learning curve. Usually it will make more sense to reimplement your sysv init script as a systemd service unit. You can eliminate hundreds of line of complicated error prone shell script with a handful of lines of systemd configuration directives that work better than the original.

You can get future posts delivered by email or good old-fashioned RSS.
TurnKey also has a presence on Google+, Twitter and Facebook.

Comments

John Carver's picture

I have a situation where I need to patch a sysv init script and then get systemd to recognize the change.  The change has been applied to /etc/init.d/lxc, but is not being recognized by 'service restart lxc'.  Apparently the change must also be applied to /usr/lib/x86_64-linux-gnu/lxc/lxc-autostart-helper, which appears to be a verbatim copy of the original /etc/init.d/lxc.

I tried copying the patched sysv init script to overwrite the lxc-autostart-helper and then ran 'systemctl daemon-reload'.  This seemed to work okay, but I doubt it is the proper way to accomplish the task.

Information is free, knowledge is acquired, but wisdom is earned.

Francis Evrard 's picture

when you type "update-rc.d lxc défaults", you should get a "systemd [1] Reload", after which things should be Ok !

Jade's picture

You noted:
"Long story short, systemd interprets the standard LSB headers in the sysv init script so it execute the sysv init script in the context of something resembling a real systemd service."

But how does one modify the script such that one can change the default "Restart" setting?

I've inherited the SysVinit scripts.  It's now running on a linux OS where systemd is the init system used.  Without having to replace the SysVinit scripts, how can I get it such that these daemons can be restarted if it crashes.  My desire is to set

Restart=on-failure

I'm guessing I need to add somethign to the LSB header???  Any idea on what that is???

Vincent Murphy's picture

Jade-

I had the same question. I am afraid we are out of luck.

The source code at https://github.com/systemd/systemd/blob/master/src/sysv-generator/sysv-g... shows that the generator always uses Restart=no , and doesn't process anything from the LSB header to change this.

Thanks to this very helpful blog post, I will take a copy of the generated service unit from /run/systemd/generated.late and change the Restart there. At least then I know I have not broken anything else in my working service.

Pages

Post new comment