systemd - One init system to rule them all...

systemctl status myApplication returns Active: failed (Result: start-limit-hit)

Situation :

systemctl status myApplication.service
● myApplication.service - myApplication Java application
   Loaded: loaded (/etc/systemd/system/myApplication.service_4.1-RELEASE; bad)
   Active: failed (Result: start-limit-hit) since Wed 2020-03-04 10:19:40 UTC; 7s ago
  Process: 103744 ExecStop=/bin/kill -15 $(systemctl status myApplication.service | awk /Main PID/ {print $3}) (code=exited, status=1/FAILURE)
  Process: 103742 ExecStart=/usr/bin/java -jar myApplication.jar --spring.config.location=classpath:/,file:application.yaml (code=exited, status=1/FAILURE)
 Main PID: 103742 (code=exited, status=1/FAILURE)

Details :

cat /etc/systemd/system/myApplication.service
[Unit]
Description=myApplication Java application

[Service]
WorkingDirectory=/path/to/myApplication
ExecStart=/usr/bin/java -jar myApplication.jar --spring.config.location=classpath:/,file:application.yaml
ExecStop=/bin/kill -15 $(systemctl status myApplication.service | awk '/Main PID/ {print $3}')
User=kevin
Group=robots
Restart=on-failure

[Install]
WantedBy=multi-user.target
systemd is configured to restart myApplication should it fail, which is exactly what happens :
  1. I ask myApplication to start with : systemctl start myApplication.service
  2. myApplication tries to start, but fails for an unknown reason
  3. systemd detects this failure and tries to start it again
  4. myApplication fails again
  5. after 5 try + fail attempts, systemd gives up and reports this start-limit-hit error. You can confirm this with :
    • journalctl -f -u myApplication.service in a terminal
    • systemctl start myApplication in another terminal
    Output : 
    Mar 04 10:34:21 myServer systemd[1]: Started myApplication Java application.
Mar 04 10:34:21 myServer systemd[1]: myApplication.service: Main process exited, code=exited, status=1/FAILURE	the startup fails
Mar 04 10:34:21 myServer kill[104802]: kill: failed to parse argument: 'status'					the ExecStop command fails too  (but this is "normal" here)
Mar 04 10:34:21 myServer systemd[1]: myApplication.service: Control process exited, code=exited status=1
Mar 04 10:34:21 myServer systemd[1]: myApplication.service: Unit entered failed state.
Mar 04 10:34:21 myServer systemd[1]: myApplication.service: Failed with result 'exit-code'.
Mar 04 10:34:21 myServer systemd[1]: myApplication.service: Service hold-off time over, scheduling restart.
Mar 04 10:34:21 myServer systemd[1]: Stopped myApplication Java application.
Mar 04 10:34:21 myServer systemd[1]: Started myApplication Java application.

(looping 4 more times)
Anyway, this means :
  • systemd works fine
  • the bug is on myApplication side

Solution :

There is no "one-size-fits-all" solution to this start-limit-hit error since it is caused by a problem on the application that is controlled by systemd, and not by systemd itself.

Things you may investigate :

  1. have a look at the myApplication logs, if any
  2. try to launch the ExecStart command manually as the User
  3. other ideas ?

Failed to set invocation ID on control group /system.slice/varnishncsa.service, ignoring: Operation not permitted

Situation :

Full error message :
systemctl status varnishncsa.service
● varnishncsa.service - Varnish Cache HTTP accelerator NCSA logging daemon
   Loaded: loaded (/etc/systemd/system/varnishncsa.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-02-20 02:16:03 CET; 1 weeks 4 days ago
 Main PID: 3309 (varnishncsa)
   CGroup: /system.slice/varnishncsa.service
		   └─3309 /usr/bin/varnishncsa [options...]

Mar 02 10:39:12 myServer systemd[1]: varnishncsa.service: Failed to set invocation ID on control group /system.slice/varnishncsa.service, ignoring: Operation not permitted

Details :

Solution :

I'm afraid there's nothing I can do to fix this, just live with it .

Are ssh.service and sshd.service the same service ?

systemctl list-unit-files | grep ssh
ssh.service			enabled
ssh@.service			static
sshd.service			enabled
ssh.socket			disabled
systemctl list-unit-files | awk '/ssh.*enabled/ {print $1}' | xargs systemctl status
● ssh.service - OpenBSD Secure Shell server
	Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
	Active: inactive (dead) since Tue 2016-12-06 15:36:52 CET; 16min ago
 Main PID: 662 (code=exited, status=0/SUCCESS)

Dec 02 17:23:45 caramba sshd[662]: Server listening on 0.0.0.0 port 22.
Dec 02 17:23:45 caramba sshd[662]: Server listening on :: port 22.
Dec 02 17:23:45 caramba systemd[1]: Reloaded OpenBSD Secure Shell server.


● ssh.service - OpenBSD Secure Shell server
	Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
	Active: inactive (dead) since Tue 2016-12-06 15:36:52 CET; 16min ago
 Main PID: 662 (code=exited, status=0/SUCCESS)

Dec 02 17:23:45 caramba sshd[662]: Server listening on 0.0.0.0 port 22.
Dec 02 17:23:45 caramba sshd[662]: Server listening on :: port 22.
Dec 02 17:23:45 caramba systemd[1]: Reloaded OpenBSD Secure Shell server.

Same PID, so ssh.service == sshd.service

Easier method :

  • grep Alias /lib/systemd/system/ssh.service
    Alias=sshd.service
  • ... and :
    ll /etc/systemd/system/ssh*
    lrwxrwxrwx 1 root root 31 Mar 27  2017 /etc/systemd/system/sshd.service -> /lib/systemd/system/ssh.service

unit configuration files

Location

Service unit configuration file
syslog /etc/systemd/system/syslog.service
unit files can be found in different directories :
/lib/systemd/system
for "official" unit files / those provided by packages. It is possible to create custom unit files here (no idea whether this is a good idea / safe practice or not...)
/etc/systemd/system
unit files (those described above) can be copied here to be customized. It is also possible to define custom unit files here (same limitation)
/etc/systemd/system/*
These directories mostly (only) contain symlinks to unit files found under /lib/systemd/system

Syntax

[Unit] section (source)
generic section
[Install] section (source)
generic section
[Service] section
for 'service' units only

Example :

Filter syslog logs entering journald by verbosity level :

  1. In /etc/systemd/system/syslog.service, add :
    LogLevelMax=err
    don't know in which section yet : Unit / Service ?
  2. systemctl daemon-reload && systemctl restart syslog; echo $?
  3. watch -n 1 -d 'journalctl -u syslog -r | head -5'

systemd configuration directives

Directive Default value Description
LogLevelMax=level debug Per-unit setting defining the maximum log level (i.e. verbosity) of log messages. level is one of syslog log levels :
  • debug : highest log level, also lowest priority messages (default)
  • info
  • notice
  • warning
  • err
  • crit
  • alert
  • emerg : lowest log level, only highest priority messages
For example, set LogLevelMax=info in order to turn off debug logging of a particularly chatty unit. Note that the configured level is applied to any log messages written by any of the processes belonging to this unit, sent via any supported logging protocol. The filtering is applied early in the logging pipeline, before any kind of further processing is done. Moreover, messages which pass through this filter successfully might still be dropped by filters applied at a later stage in the logging subsystem. For example, MaxLevelStore might prohibit messages of higher log levels to be stored on disk, even though the per-unit LogLevelMax permitted it to be processed.

systemd glossary

active
a service unit is currently running
inactive
a service unit is currently disabled but may get started (i.e. become active) if something attempts to make use of the service.
  • Think of systemd along the same lines as xinetd : it can manage your services for you and start them up, on demand when needed. So while the services are "off" they're in the inactive state, but when started, they can become active.
  • This state can also occur when a service unit has been enabled but not yet manually started. So the service lays "dormant" in the stopped or failed state until
    • either the service is manually started
    • or the system goes through a reboot, which would cause the service to become active due to its enablement
(source)
enabled / disabled
a service unit is configured to start / not to start when the system boots (source)
target
unit
object that is relevant for system boot-up and maintenance. The majority of units are configured in unit configuration files. Units can be :
  • service units : to manage daemons
  • socket units : for local IPC or network sockets
  • target units : to group units, or synchronize stuff during boot-up
  • device units
  • mount units : to control mount points in the file system
  • timer units : trigger activation of other units based on timers
  • path units : may be used to activate other services when file system objects change or are modified
  • slice units :
  • units : may be used to group units which manage system processes (such as service and scope units) in a hierarchical tree for resource management purposes
  • and more...
(source : man systemd + search Units)
unit configuration file
file that encodes information about a service, a socket, a device, a mount point, an automount point, a swap file or partition, ... : anything managed by systemd and aka a unit.
(details)

systemd

This article is about :

Instead of runlevels, systemd allows you to create different states, which gives you a flexible mechanism for creating different configurations to boot into. These states are composed of multiple unit files bundled into targets. Targets have nice descriptive names instead of numbers. Unit files control services, devices, sockets, and mounts.

With systemd : 
ll /etc/systemd/system/multi-user.target.wants/

lrwxrwxrwx 1 root root 35 nov.	23 21:01 anacron.service -> /lib/systemd/system/anacron.service
lrwxrwxrwx 1 root root 31 nov.	23 21:01 atd.service -> /lib/systemd/system/atd.service
lrwxrwxrwx 1 root root 32 nov.	23 20:24 cron.service -> /lib/systemd/system/cron.service
lrwxrwxrwx 1 root root 40 nov.	23 21:02 ModemManager.service -> /lib/systemd/system/ModemManager.service
lrwxrwxrwx 1 root root 42 nov.	23 21:02 NetworkManager.service -> /lib/systemd/system/NetworkManager.service
lrwxrwxrwx 1 root root 31 nov.	23 21:02 ssh.service -> /lib/systemd/system/ssh.service
...

List existing daemons (aka units) :

Not sure if this is a general rule, but many systemd services are now called "daemon.service" : ssh.service, tomcat8.service, ...

I don't know (yet!) why I can observe this (explicitly with MySQL, not tested with others) :
systemctl list-unit-files | grep -i mysql
(nothing)
systemctl list-units | grep -i mysql
mysql.service		loaded active running	LSB: Start and stop the mysql database server daemon
No idea whether :
  • this is related to MySQL
  • systemctl list-units and systemctl list-unit-files read different sources then report different results
  • ...

Get status of a daemon, systemd vs Upstart vs SysVinit :

systemd :
systemctl status daemon.service
It's ok if this says : Active: active (exited). This means this is not really a daemon listening, but the task itself started + ran + exited normally.
Upstart :
service daemon status
SysVinit :
/etc/init.d/daemon status
http://forums.fedoraforum.org/showthread.php?t=261945
'service' operates on the files in /etc/init.d and was used in conjunction with the old init system.
'systemctl' operates on the files in /lib/systemd
If there is a file for your service in /lib/systemd, it will use that first. Otherwise it will fall back to the file in /etc/init.d

In theory service is supposed to be linked to systemctl but in the F16 branch it looks like they are starting to disable that.

Is a daemon enabled (source) ?

systemctl is-enabled varnish

Disable a daemon (i.e. prevent it from being started at boot time) :

systemctl stop unwantedDaemon && systemctl disable unwantedDaemon