systemctl - description, flags, examples and more

mail

systemctl daemon-reload

systemctl daemon-reload reloads the systemd manager configuration :
  1. rerun all generators
  2. reload all unit files
  3. recreate the entire dependency tree
source / details : After editing the unit file of the myService service, you have to :
  1. make systemd aware of the updated version of this unit file :
    systemctl daemon-reload
  2. reload the service itself :
    systemctl restart myService.service
mail

systemctl status myApplication returns Active: failed (Result: start-limit-hit)

Situation

systemctl status myApplication.service
● myApplication.service - myApplication Java application
   Loaded: loaded (/etc/systemd/system/myApplication.service_4.1-RELEASE; bad)
   Active: failed (Result: start-limit-hit) since Wed 2020-03-04 10:19:40 UTC; 7s ago
  Process: 103744 ExecStop=/bin/kill -15 $(systemctl status myApplication.service | awk /Main PID/ {print $3}) (code=exited, status=1/FAILURE)
  Process: 103742 ExecStart=/usr/bin/java -jar myApplication.jar --spring.config.location=classpath:/,file:application.yaml (code=exited, status=1/FAILURE)
 Main PID: 103742 (code=exited, status=1/FAILURE)

Details

cat /etc/systemd/system/myApplication.service
[Unit]
Description=myApplication Java application

[Service]
WorkingDirectory=/path/to/myApplication
ExecStart=/usr/bin/java -jar myApplication.jar --spring.config.location=classpath:/,file:application.yaml
ExecStop=/bin/kill -15 $(systemctl status myApplication.service | awk '/Main PID/ {print $3}')
User=kevin
Group=robots
Restart=on-failure

[Install]
WantedBy=multi-user.target
systemd is configured to restart myApplication should it fail, which is exactly what happens :
  1. I ask myApplication to start with : systemctl start myApplication.service
  2. myApplication tries to start, but fails for an unknown reason
  3. systemd detects this failure and tries to start it again
  4. myApplication fails again
  5. after 5 try + fail attempts, systemd gives up and reports this start-limit-hit error. You can confirm this with :
    • journalctl -f -u myApplication.service in a terminal
    • systemctl start myApplication in another terminal
    Output : 
    Mar 04 10:34:21 myServer systemd[1]: Started myApplication Java application.
Mar 04 10:34:21 myServer systemd[1]: myApplication.service: Main process exited, code=exited, status=1/FAILURE	the startup fails
Mar 04 10:34:21 myServer kill[104802]: kill: failed to parse argument: 'status'					the ExecStop command fails too  (but this is "normal" here)
Mar 04 10:34:21 myServer systemd[1]: myApplication.service: Control process exited, code=exited status=1
Mar 04 10:34:21 myServer systemd[1]: myApplication.service: Unit entered failed state.
Mar 04 10:34:21 myServer systemd[1]: myApplication.service: Failed with result 'exit-code'.
Mar 04 10:34:21 myServer systemd[1]: myApplication.service: Service hold-off time over, scheduling restart.
Mar 04 10:34:21 myServer systemd[1]: Stopped myApplication Java application.
Mar 04 10:34:21 myServer systemd[1]: Started myApplication Java application.

(looping 4 more times)
Anyway, this means :
  • systemd works fine
  • the bug is on myApplication side

Solution

There is no "one-size-fits-all" solution to this start-limit-hit error since it is caused by a problem on the application that is controlled by systemd, and not by systemd itself.

Things you may investigate :

  1. have a look at the myApplication logs, if any
  2. try to launch the ExecStart command manually as the User
  3. other ideas ?
mail

systemctl

Usage

Utility to control "stuff managed by systemd" a.k.a units

Flags

Flag Usage
-l --full Do not ellipsize unit names, process tree entries, journal output, or truncate unit descriptions in the output of status, list-units, list-jobs, and list-timers
--now
  • systemctl enable --now unit = systemctl enable unit && systemctl start unit
  • systemctl disable --now unit = systemctl disable unit && systemctl stop unit
daemon-reload Reload systemd's configuration, see dedicated article
not be confused with reload
disable unit do NOT start unit at boot time.
  1. delete symlinks (those created by enable + symlinks created manually)
  2. reload the system manager configuration
  • this does NOT stop the corresponding services. To do so, use --now
  • it can be silenced with --quiet
enable unit start unit at boot time. This actually :
  1. creates a set of symlinks, according to the [Install] section of the indicated unit file
  2. reloads the system manager configuration (in a way equivalent to daemon-reload) in order to ensure the changes are taken into account immediately
Possible causes of : 
systemctl enable unit
Failed to execute operation: Invalid argument
  • unit is unknown to systemd until you run systemctl daemon-reload
  • unit is a symlink (but it works with hardlinks, though)
  • a unit file must be named whatever.service
list-dependencies [options] Recursively show dependencies of the specified unit. Example :
systemctl list-dependencies graphical.target
list-unit-files [options] List unit files installed on the system and their enablement state : enabled / disabled / masked / static / generated / .... Example :
systemctl list-unit-files --type=service
reload pattern Asks all units listed on the command line to reload their configuration (i.e. ask daemons managed by systemd to reload their own configuration)
This will reload the service-specific configuration, not the unit configuration file of systemd. If you want systemd to reload the configuration file of a unit, use daemon-reload. In other words: for the example case of Apache, this will reload Apache's httpd.conf in the web server, not the apache.service systemd unit file.
not be confused with daemon-reload
restart pattern Stop then start matching units. Any unit not already running will be started
show
  • systemctl show unit : show properties of unit
  • systemctl show jobId : show properties of jobId
  • systemctl show : show properties of systemd itself
  • --show is intended to be used whenever computer-parsable output is required. For human-readable output, use status.
  • It is possible to query specific properties :
    systemctl show mysql --property=ActiveEnterTimestamp
status unit Show terse runtime status information about unit, followed by most recent log data from the journal.
For a different output format :
systemctl status --output=json-pretty nginx

Example

Start / stop / status of a daemon :

systemctl start|stop|status daemon.service

While debugging, it may be useful to prefix systemctl invocation with date :

date; systemctl start docker.service
so that it's easier to identify journalctl entries if the operation failed.

View the unit configuration file of a service :

systemctl cat docker.service

Get the uptime of a service :

systemctl status mysql

	● mysql.service - MySQL Community Server
	   Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
	   Active: active (running) since Fri 2019-11-29 15:10:38 UTC; 2 days ago
	  Process: 1111 ExecStartPost=/usr/share/mysql/mysql-systemd-start post (code=exited, status=0/SUCCESS)
	  Process: 1050 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
	 Main PID: 1110 (mysqld)

ps -o etime= 1110
	 2-17:50:46

==========================================
systemctl show mysql --property=MainPID | cut -d '=' -f 2
ps -o etime= $(systemctl show mysql --property=MainPID | cut -d '=' -f 2)
	 2-18:44:12

==========================================
systemctl show mysql --property=ActiveEnterTimestamp
	ActiveEnterTimestamp=Fri 2019-11-29 15:10:38 UTC

==========================================
ps -o lstart= $(systemctl show mysql --property=MainPID | cut -d '=' -f 2)
	Fri Nov 29 15:10:36 2019

echo $(($(date -d"now" +%s) - $(date -d"Fri Nov 29 15:10:36 2019" +%s)))


echo $(($(date -d"now" +%s) - $(date -d"$(ps -o lstart= $(systemctl show mysql --property=MainPID | cut -d '=' -f 2))" +%s)))

	==> uptime in seconds