systemd - One init system to rule them all...

mail

Failed to set invocation ID on control group /system.slice/varnishncsa.service, ignoring: Operation not permitted

Situation

Full error message :
systemctl status varnishncsa.service
● varnishncsa.service - Varnish Cache HTTP accelerator NCSA logging daemon
   Loaded: loaded (/etc/systemd/system/varnishncsa.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-02-20 02:16:03 CET; 1 weeks 4 days ago
 Main PID: 3309 (varnishncsa)
   CGroup: /system.slice/varnishncsa.service
		   └─3309 /usr/bin/varnishncsa [options...]

Mar 02 10:39:12 myServer systemd[1]: varnishncsa.service: Failed to set invocation ID on control group /system.slice/varnishncsa.service, ignoring: Operation not permitted

Details

Solution

I'm afraid there's nothing I can do to fix this, just live with it .
mail

Are ssh.service and sshd.service the same service ?

systemctl list-unit-files | grep ssh
ssh.service			enabled
ssh@.service			static
sshd.service			enabled
ssh.socket			disabled
systemctl list-unit-files | awk '/ssh.*enabled/ {print $1}' | xargs systemctl status
● ssh.service - OpenBSD Secure Shell server
	Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
	Active: inactive (dead) since Tue 2016-12-06 15:36:52 CET; 16min ago
 Main PID: 662 (code=exited, status=0/SUCCESS)

Dec 02 17:23:45 caramba sshd[662]: Server listening on 0.0.0.0 port 22.
Dec 02 17:23:45 caramba sshd[662]: Server listening on :: port 22.
Dec 02 17:23:45 caramba systemd[1]: Reloaded OpenBSD Secure Shell server.


● ssh.service - OpenBSD Secure Shell server
	Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
	Active: inactive (dead) since Tue 2016-12-06 15:36:52 CET; 16min ago
 Main PID: 662 (code=exited, status=0/SUCCESS)

Dec 02 17:23:45 caramba sshd[662]: Server listening on 0.0.0.0 port 22.
Dec 02 17:23:45 caramba sshd[662]: Server listening on :: port 22.
Dec 02 17:23:45 caramba systemd[1]: Reloaded OpenBSD Secure Shell server.

Same PID, so ssh.service == sshd.service

Easier method :

  • grep Alias /lib/systemd/system/ssh.service
    Alias=sshd.service
  • ... and :
    ll /etc/systemd/system/ssh*
    lrwxrwxrwx 1 root root 31 Mar 27  2017 /etc/systemd/system/sshd.service -> /lib/systemd/system/ssh.service
mail

unit configuration files

unit files can be found in different directories :

Locations also vary by distribution families : you won't find things at the same place on Red Hatoids and on Debianoids, making things even more confusing .
For full details about these locations, see : There are even more directories involved (/run/, ) and precedence rules between all of them.

/lib/systemd vs /usr/lib/systemd :

  • Are there the "same" files ?
    ll {,/usr}/lib/systemd/systemd-journald
    -rwxr-xr-x 1 root root 162176 Nov 21 21:57 /lib/systemd/systemd-journald*
-rwxr-xr-x 1 root root 162176 Nov 21 21:57 /usr/lib/systemd/systemd-journald*
    Both files exist and look similar.
  • Are these actually identical ?
    md5sum {,/usr}/lib/systemd/systemd-journald
    9e5f643ae1fd0586021a49e5edef8f82  /lib/systemd/systemd-journald
9e5f643ae1fd0586021a49e5edef8f82  /usr/lib/systemd/systemd-journald
    Yes !
  • Are all files identical ?
    for i in $(find /lib/systemd -type f); do md5sum "$i" "/usr$i"; done | sed -r 's| /usr| |' | uniq -c | awk '$1 != 2'
    If this returns nothing, it means all actual files found under /lib/systemd and their /usr/lib/systemd counterparts are identical.

What's the purpose / difference between the /etc and /lib-based directories (source) ?

  • /lib-based directories (/lib and /usr/lib) :
    • for unit files provided by packages during installation
    • Do not edit these unless you are a distro maintainer or you enjoy your files getting overwritten on each package update. Workarounds :
      • copy the unit file into /etc/systemd and edit this version as you like
      • use a drop-in : /serviceName.d/custom.conf (details : man 5 systemd.unit + search drop-in)
      • suggested : systemctl edit unitFileToEdit, which does the drop-in file magic automatically!
  • /etc-based directories :
    • to alter a "default" unit file, copy it there from /lib/ then edit it
    • custom unit files can be created here

/etc-based files trees

on Ubuntu
/etc/systemd/
	*conf										regular conf files
	network/									is empty
	system/
		<foo>.target.wants/
			<bar>.service -> /lib/systemd/system/<bar>.service
		<baz>.service -> /lib/systemd/system/<bam>.service
		snap-*.mount
	user/
		<serviceName>.target.wants/						full of these
			<zap>.service -> /usr/lib/systemd/user/<zap>.service
on Red Hat 7
/etc/systemd/
	*conf										regular conf files
	system/
		<foo>.target.wants/
			<bar>.service ->  /usr/lib/systemd/system/<bar>.service

		<baz>.service -> /usr/lib/systemd/system/<bam>.service			those installed by packages
		<zap>.service -> /opt/docker/...					custom stuff made by ourselves
	user/										is empty

Syntax

[Unit] section (source)

generic section

[Install] section (source)

generic section

[Service] section

for 'service' units only

Example

Filter syslog logs entering journald by verbosity level :

  1. In /etc/systemd/system/syslog.service, add :
    LogLevelMax=err
    don't know in which section yet : Unit / Service ?
  2. systemctl daemon-reload && systemctl restart syslog; echo $?
  3. watch -n 1 -d 'journalctl -u syslog -r | head -5'
mail

systemd configuration directives

Directive Default value Description
LogLevelMax=level debug Per-unit setting defining the maximum log level (i.e. verbosity) of log messages. level is one of syslog log levels :
  • debug : highest log level, also lowest priority messages (default)
  • info
  • notice
  • warning
  • err
  • crit
  • alert
  • emerg : lowest log level, only highest priority messages
For example, set LogLevelMax=info in order to turn off debug logging of a particularly chatty unit. Note that the configured level is applied to any log messages written by any of the processes belonging to this unit, sent via any supported logging protocol. The filtering is applied early in the logging pipeline, before any kind of further processing is done. Moreover, messages which pass through this filter successfully might still be dropped by filters applied at a later stage in the logging subsystem. For example, MaxLevelStore might prohibit messages of higher log levels to be stored on disk, even though the per-unit LogLevelMax permitted it to be processed.
mail

systemd glossary

active
a service unit is currently running
inactive
a service unit is currently disabled but may get started (i.e. become active) if something attempts to make use of the service.
  • Think of systemd along the same lines as xinetd : it can manage your services for you and start them up, on demand when needed. So while the services are "off" they're in the inactive state, but when started, they can become active.
  • This state can also occur when a service unit has been enabled but not yet manually started. So the service lays "dormant" in the stopped or failed state until
    • either the service is manually started
    • or the system goes through a reboot, which would cause the service to become active due to its enablement
(source)
enabled / disabled
a service unit is configured to start / not to start when the system boots
target
  • unit configuration files whose name end in .target encode information about target units of systemd
  • targets are used for grouping units via dependencies and as well-known synchronization points during start-up.
  • among other things, target units are a more flexible replacement for SysV runlevels in the classic SysV init system.
unit
object that is relevant for system boot-up and maintenance. The majority of units are configured in unit configuration files. Units can be :
  • service units : to manage daemons
  • socket units : for local IPC or network sockets
  • target units : to group units, or synchronize stuff during boot-up
  • device units
  • mount units : to control mount points in the file system
  • timer units : trigger activation of other units based on timers
  • path units : may be used to activate other services when file system objects change or are modified
  • slice units :
  • units : may be used to group units which manage system processes (such as service and scope units) in a hierarchical tree for resource management purposes
  • and more...
(source : man systemd + search Units)
unit configuration file
file that encodes information about a service, a socket, a device, a mount point, an automount point, a swap file or partition, ... : anything managed by systemd and aka a unit.
(details)
mail

systemd

This article is about :

Instead of runlevels, systemd allows you to create different states, which gives you a flexible mechanism for creating different configurations to boot into. These states are composed of multiple unit files bundled into targets. Targets have nice descriptive names instead of numbers. Unit files control services, devices, sockets, and mounts.

With systemd : 
ll /etc/systemd/system/multi-user.target.wants/

lrwxrwxrwx 1 root root 35 nov.	23 21:01 anacron.service -> /lib/systemd/system/anacron.service
lrwxrwxrwx 1 root root 31 nov.	23 21:01 atd.service -> /lib/systemd/system/atd.service
lrwxrwxrwx 1 root root 32 nov.	23 20:24 cron.service -> /lib/systemd/system/cron.service
lrwxrwxrwx 1 root root 40 nov.	23 21:02 ModemManager.service -> /lib/systemd/system/ModemManager.service
lrwxrwxrwx 1 root root 42 nov.	23 21:02 NetworkManager.service -> /lib/systemd/system/NetworkManager.service
lrwxrwxrwx 1 root root 31 nov.	23 21:02 ssh.service -> /lib/systemd/system/ssh.service
...

List existing daemons (aka units) :

Not sure if this is a general rule, but many systemd services are now called "daemon.service" : ssh.service, tomcat8.service, ...

I don't know (yet!) why I can observe this (explicitly with MySQL, not tested with others) :
systemctl list-unit-files | grep -i mysql
(nothing)
systemctl list-units | grep -i mysql
mysql.service		loaded active running	LSB: Start and stop the mysql database server daemon
No idea whether :
  • this is related to MySQL
  • systemctl list-units and systemctl list-unit-files read different sources then report different results
  • ...

Is a daemon enabled ?

systemctl is-enabled varnish

Disable a daemon (i.e. prevent it from being started at boot time) :

systemctl stop unwantedDaemon && systemctl disable unwantedDaemon

Get status of a daemon, systemd vs Upstart vs SysVinit :

systemd :
systemctl status daemon.service
It's ok if this says : Active: active (exited). This means this is not really a daemon listening, but the task itself started + ran + exited normally.
Upstart :
service daemon status
SysVinit :
/etc/init.d/daemon status
http://forums.fedoraforum.org/showthread.php?t=261945
'service' operates on the files in /etc/init.d and was used in conjunction with the old init system.
'systemctl' operates on the files in /lib/systemd
If there is a file for your service in /lib/systemd, it will use that first. Otherwise it will fall back to the file in /etc/init.d

In theory service is supposed to be linked to systemctl but in the F16 branch it looks like they are starting to disable that.