systemd

systemctl status myApplication.service

● myApplication.service - myApplication Java application
   Loaded: loaded (/etc/systemd/system/myApplication.service_4.1-RELEASE; bad)
   Active: failed (Result: start-limit-hit) since Wed 2020-03-04 10:19:40 UTC; 7s ago
  Process: 103744 ExecStop=/bin/kill -15 $(systemctl status myApplication.service | awk /Main PID/ {print $3}) (code=exited, status=1/FAILURE)
  Process: 103742 ExecStart=/usr/bin/java -jar myApplication.jar --spring.config.location=classpath:/,file:application.yaml (code=exited, status=1/FAILURE)
 Main PID: 103742 (code=exited, status=1/FAILURE)

cat /etc/systemd/system/myApplication.service

[Unit]
Description=myApplication Java application

[Service]
WorkingDirectory=/path/to/myApplication
ExecStart=/usr/bin/java -jar myApplication.jar --spring.config.location=classpath:/,file:application.yaml
ExecStop=/bin/kill -15 $(systemctl status myApplication.service | awk '/Main PID/ {print $3}')
User=kevin
Group=robots
Restart=on-failure

[Install]
WantedBy=multi-user.target

systemd is configured to restart myApplication should it fail, which is exactly what happens :

I ask myApplication to start with : systemctl start myApplication.service
myApplication tries to start, but fails for an unknown reason
systemd detects this failure and tries to start it again
myApplication fails again

after 5 try + fail attempts, systemd gives up and reports this start-limit-hit error. You can confirm this with :

journalctl -f -u myApplication.service in a terminal
systemctl start myApplication in another terminal

Output :

Mar 04 10:34:21 myServer systemd[1]: Started myApplication Java application.
Mar 04 10:34:21 myServer systemd[1]: myApplication.service: Main process exited, code=exited, status=1/FAILURE	the startup fails
Mar 04 10:34:21 myServer kill[104802]: kill: failed to parse argument: 'status'					the ExecStop command fails too  (but this is "normal" here)
Mar 04 10:34:21 myServer systemd[1]: myApplication.service: Control process exited, code=exited status=1
Mar 04 10:34:21 myServer systemd[1]: myApplication.service: Unit entered failed state.
Mar 04 10:34:21 myServer systemd[1]: myApplication.service: Failed with result 'exit-code'.
Mar 04 10:34:21 myServer systemd[1]: myApplication.service: Service hold-off time over, scheduling restart.
Mar 04 10:34:21 myServer systemd[1]: Stopped myApplication Java application.
Mar 04 10:34:21 myServer systemd[1]: Started myApplication Java application.

(looping 4 more times)

Anyway, this means :

systemd works fine
the bug is on myApplication side

There is no "one-size-fits-all" solution to this start-limit-hit error since it is caused by a problem on the application that is controlled by systemd, and not by systemd itself.

Things you may investigate :

have a look at the myApplication logs, if any
try to launch the ExecStart command manually as the User
other ideas ?

Full error message :

systemctl status varnishncsa.service

● varnishncsa.service - Varnish Cache HTTP accelerator NCSA logging daemon
   Loaded: loaded (/etc/systemd/system/varnishncsa.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-02-20 02:16:03 CET; 1 weeks 4 days ago
 Main PID: 3309 (varnishncsa)
   CGroup: /system.slice/varnishncsa.service
		   └─3309 /usr/bin/varnishncsa [options...]

Mar 02 10:39:12 myServer systemd[1]: varnishncsa.service: Failed to set invocation ID on control group /system.slice/varnishncsa.service, ignoring: Operation not permitted

This has nothing to do with Varnish and Varnish NCSA. A similar error message appears with other softwares.
This is related to systemd itself and caused by the kernel version (source)
In my specific case, where myServer is an OpenVZ instance, the kernel version is way too old, and incompatible with systemd (source)

I'm afraid there's nothing I can do to fix this, just live with it .

systemctl list-unit-files | grep ssh

ssh.service			enabled
ssh@.service			static
sshd.service			enabled
ssh.socket			disabled

systemctl list-unit-files | awk '/ssh.*enabled/ {print $1}' | xargs systemctl status

● ssh.service - OpenBSD Secure Shell server
	Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
	Active: inactive (dead) since Tue 2016-12-06 15:36:52 CET; 16min ago
 Main PID: 662 (code=exited, status=0/SUCCESS)

Dec 02 17:23:45 caramba sshd[662]: Server listening on 0.0.0.0 port 22.
Dec 02 17:23:45 caramba sshd[662]: Server listening on :: port 22.
Dec 02 17:23:45 caramba systemd[1]: Reloaded OpenBSD Secure Shell server.


● ssh.service - OpenBSD Secure Shell server
	Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
	Active: inactive (dead) since Tue 2016-12-06 15:36:52 CET; 16min ago
 Main PID: 662 (code=exited, status=0/SUCCESS)

Dec 02 17:23:45 caramba sshd[662]: Server listening on 0.0.0.0 port 22.
Dec 02 17:23:45 caramba sshd[662]: Server listening on :: port 22.
Dec 02 17:23:45 caramba systemd[1]: Reloaded OpenBSD Secure Shell server.

Same PID, so ssh.service == sshd.service

Easier method :

grep Alias /lib/systemd/system/ssh.service
```
Alias=sshd.service
```

... and :

ll /etc/systemd/system/ssh*

lrwxrwxrwx 1 root root 31 Mar 27  2017 /etc/systemd/system/sshd.service -> /lib/systemd/system/ssh.service

Location

Service	unit configuration file
syslog	/etc/systemd/system/syslog.service

unit files can be found in different directories :

/lib/systemd/system: for "official" unit files / those provided by packages. It is possible to create custom unit files here (no idea whether this is a good idea / safe practice or not...)
/etc/systemd/system: unit files (those described above) can be copied here to be customized. It is also possible to define custom unit files here (same limitation)
/etc/systemd/system/*: These directories mostly (only) contain symlinks to unit files found under /lib/systemd/system

Syntax

`[Unit]` section (source)

generic section

`[Install]` section (source)

generic section

`[Service]` section

for 'service' units only

Filter syslog logs entering journald by verbosity level :

In /etc/systemd/system/syslog.service, add :
```
LogLevelMax=err
```
don't know in which section yet : Unit / Service ?
systemctl daemon-reload && systemctl restart syslog; echo $?
watch -n 1 -d 'journalctl -u syslog -r | head -5'

Directive	Default value	Description
LogLevelMax=level	debug	Per-unit setting defining the maximum log level (i.e. verbosity) of log messages. level is one of syslog log levels : debug : highest log level, also lowest priority messages (default) info notice warning err crit alert emerg : lowest log level, only highest priority messages For example, set LogLevelMax=info in order to turn off debug logging of a particularly chatty unit. Note that the configured level is applied to any log messages written by any of the processes belonging to this unit, sent via any supported logging protocol. The filtering is applied early in the logging pipeline, before any kind of further processing is done. Moreover, messages which pass through this filter successfully might still be dropped by filters applied at a later stage in the logging subsystem. For example, MaxLevelStore might prohibit messages of higher log levels to be stored on disk, even though the per-unit LogLevelMax permitted it to be processed. (see commit details) this appeared in systemd 2.36 Jessie tops at systemd 2.15

active

a service unit is currently running

inactive

a service unit is currently disabled but may get started (i.e. become active) if something attempts to make use of the service.

Think of systemd along the same lines as xinetd : it can manage your services for you and start them up, on demand when needed. So while the services are "off" they're in the inactive state, but when started, they can become active.
This state can also occur when a service unit has been enabled but not yet manually started. So the service lays "dormant" in the stopped or failed state until
- either the service is manually started
- or the system goes through a reboot, which would cause the service to become active due to its enablement

(source)

enabled / disabled

a service unit is configured to start / not to start when the system boots (source)

target

unit

object that is relevant for system boot-up and maintenance. The majority of units are configured in unit configuration files. Units can be :

service units : to manage daemons
socket units : for local IPC or network sockets
target units : to group units, or synchronize stuff during boot-up
device units
mount units : to control mount points in the file system
timer units : trigger activation of other units based on timers
path units : may be used to activate other services when file system objects change or are modified
slice units :
units : may be used to group units which manage system processes (such as service and scope units) in a hierarchical tree for resource management purposes
and more...

(source : man systemd + search Units)

unit configuration file

file that encodes information about a service, a socket, a device, a mount point, an automount point, a swap file or partition, ... : anything managed by systemd and aka a unit.
(details)

This article is about :

the "old" way of machine states to boot into (the runlevels)
the "old" SysVInit
Upstart
the startup scripts
and the new way of managing these : systemd

Instead of runlevels, systemd allows you to create different states, which gives you a flexible mechanism for creating different configurations to boot into. These states are composed of multiple unit files bundled into targets. Targets have nice descriptive names instead of numbers. Unit files control services, devices, sockets, and mounts.

With systemd :

no more static runlevels, but several configuration states (kind of "dynamic runlevels") to boot into
states are defined with unit files
unit files are bundled in targets

ll /etc/systemd/system/multi-user.target.wants/

lrwxrwxrwx 1 root root 35 nov.	23 21:01 anacron.service -> /lib/systemd/system/anacron.service
lrwxrwxrwx 1 root root 31 nov.	23 21:01 atd.service -> /lib/systemd/system/atd.service
lrwxrwxrwx 1 root root 32 nov.	23 20:24 cron.service -> /lib/systemd/system/cron.service
lrwxrwxrwx 1 root root 40 nov.	23 21:02 ModemManager.service -> /lib/systemd/system/ModemManager.service
lrwxrwxrwx 1 root root 42 nov.	23 21:02 NetworkManager.service -> /lib/systemd/system/NetworkManager.service
lrwxrwxrwx 1 root root 31 nov.	23 21:02 ssh.service -> /lib/systemd/system/ssh.service
...

List existing daemons (aka units) :

systemctl list-units
systemctl list-unit-files | grep ssh
You can get a pretty good hint on a "service" name with : find /lib/systemd/ -name "*ssh*"
systemctl list-unit-files --type=service

Not sure if this is a general rule, but many systemd services are now called "daemon.service" : ssh.service, tomcat8.service, ...

I don't know (yet!) why I can observe this (explicitly with MySQL, not tested with others) :

systemctl list-unit-files | grep -i mysql

(nothing)

systemctl list-units | grep -i mysql

mysql.service		loaded active running	LSB: Start and stop the mysql database server daemon

No idea whether :

this is related to MySQL
systemctl list-units and systemctl list-unit-files read different sources then report different results
...

Is a daemon enabled ?

systemctl is-enabled varnish

Disable a daemon (i.e. prevent it from being started at boot time) :

systemctl stop unwantedDaemon && systemctl disable unwantedDaemon

Get status of a daemon, systemd vs Upstart vs SysVinit :

systemd :: systemctl status daemon.service

It's ok if this says : Active: active (exited). This means this is not really a daemon listening, but the task itself started + ran + exited normally.
Upstart :: service daemon status
SysVinit :: /etc/init.d/daemon status

http://forums.fedoraforum.org/showthread.php?t=261945
'service' operates on the files in /etc/init.d and was used in conjunction with the old init system.
'systemctl' operates on the files in /lib/systemd
If there is a file for your service in /lib/systemd, it will use that first. Otherwise it will fall back to the file in /etc/init.d

In theory service is supposed to be linked to systemctl but in the F16 branch it looks like they are starting to disable that.

Httqm's Docs

systemctl status myApplication returns Active: failed (Result: start-limit-hit)

Situation :

Details :

Solution :

Things you may investigate :

Failed to set invocation ID on control group /system.slice/varnishncsa.service, ignoring: Operation not permitted

Situation :

Details :

Solution :

Are ssh.service and sshd.service the same service ?

Easier method :

unit configuration files

Location

unit files can be found in different directories :

Syntax

`[Unit]` section (source)

`[Install]` section (source)

`[Service]` section

Example :

Filter syslog logs entering journald by verbosity level :

systemd configuration directives

systemd glossary

List existing daemons (aka units) :

Is a daemon enabled ?

Disable a daemon (i.e. prevent it from being started at boot time) :

Get status of a daemon, systemd vs Upstart vs SysVinit :

systemctl status myApplication returns Active: failed (Result: start-limit-hit)

Situation :

Details :

Solution :

Things you may investigate :

Failed to set invocation ID on control group /system.slice/varnishncsa.service, ignoring: Operation not permitted

Situation :

Details :

Solution :

Are ssh.service and sshd.service the same service ?

Easier method :

unit configuration files

Location

unit files can be found in different directories :

Syntax

[Unit] section (source)

[Install] section (source)

[Service] section

Example :

Filter syslog logs entering journald by verbosity level :

systemd configuration directives

systemd glossary

systemd

List existing daemons (aka units) :

Is a daemon enabled ?

Disable a daemon (i.e. prevent it from being started at boot time) :

Get status of a daemon, systemd vs Upstart vs SysVinit :

`[Unit]` section (source)

`[Install]` section (source)

`[Service]` section