systemd and Managing Services

Analogy categoryAnalogies

systemd is the init system and service manager used by virtually every major Linux distribution today (Ubuntu, Debian, RHEL/CentOS/Fedora, Arch, and more), running as PID 1 — the first userspace process the kernel starts, responsible for bringing the rest of the system up and supervising it thereafter. It replaced the older SysV init and Upstart systems with a model built around 'units': declarative configuration files describing services, mount points, devices, sockets, timers, and more, along with explicit dependency relationships between them. systemd parallelizes startup where dependencies allow, restarts failed services automatically when configured to, and centralizes logging through journald, making it the primary interface for managing what runs on a modern Linux server.

🏏

Cricket analogy: Like the team manager who's first on the ground and responsible for organizing every player's arrival, systemd replaced the sequential roll-call of old SysV init with unit files letting players warm up in parallel instead of one at a time.

Unit Files and systemctl Basics

A service unit is a plain-text file, typically ending in .service, that describes how to start, stop, and supervise a program. Distribution-provided unit files usually live in /usr/lib/systemd/system/ (or /lib/systemd/system/), while custom or overriding units belong in /etc/systemd/system/, which takes precedence. The systemctl command is the single tool for interacting with systemd: starting and stopping units immediately, enabling or disabling them so they do (or don't) start automatically at boot, and checking their current status. Critically, 'start/stop' and 'enable/disable' are independent — a service can be enabled (will start at next boot) but not currently running, or running right now but not enabled (won't survive a reboot).

🏏

Cricket analogy: Like a franchise's official player contract that a specific team can override with a side agreement, a player can be under contract for next season without playing today, or playing today on a temporary deal without being under contract at all.

bash

# Start a service immediately (does not affect boot-time behavior)
sudo systemctl start nginx

# Stop a running service
sudo systemctl stop nginx

# Restart (stop then start)
sudo systemctl restart nginx

# Reload config without a full restart (if the service supports it)
sudo systemctl reload nginx

# Enable so it starts automatically at every boot (doesn't start it now)
sudo systemctl enable nginx

# Enable AND start in one command
sudo systemctl enable --now nginx

# Disable from starting at boot
sudo systemctl disable nginx

# Check current status, recent log lines, and whether it's enabled
systemctl status nginx

# Quick boolean checks, useful in scripts
systemctl is-active nginx
systemctl is-enabled nginx

# List all currently loaded units
systemctl list-units --type=service

# List services that failed to start
systemctl --failed

Writing a Custom Service Unit

Custom unit files typically define three sections: [Unit] for metadata and dependency ordering (Description, After, Requires), [Service] for how the process itself should be run (ExecStart, Restart policy, the user to run as, working directory), and [Install] for how it hooks into boot targets (WantedBy). After creating or editing a unit file, you must run systemctl daemon-reload so systemd re-reads its unit definitions from disk — a very common mistake is editing a unit file and then wondering why systemctl restart doesn't pick up the changes, when the fix is simply the missing daemon-reload step.

🏏

Cricket analogy: Like updating a team's official strategy sheet but the umpire's rulebook copy isn't updated until someone hands over the revised version, forgetting systemctl daemon-reload after editing a unit file is why the new batting order doesn't take effect.

bash

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application API Server
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/venv/bin/python /opt/myapp/server.py
Restart=on-failure
RestartSec=5
Environment="APP_ENV=production"
EnvironmentFile=-/etc/myapp/env

[Install]
WantedBy=multi-user.target

After saving that file: sudo systemctl daemon-reload, then sudo systemctl enable --now myapp to enable it at boot and start it immediately. The Restart=on-failure directive tells systemd to automatically restart the service if it exits with a non-zero code or is killed by a signal, with RestartSec controlling the delay between attempts — this is one of systemd's biggest practical advantages over older init systems, giving you crash resilience without any custom supervisor script.

🏏

Cricket analogy: Like a captain who reviews the new fielding plan then immediately puts it into effect for the current over, setting a standing instruction that a fielder who drops a catch gets sent back in after a short breather rather than the team collapsing.

systemd 'targets' are the rough equivalent of old SysV runlevels but are more flexible — multi-user.target (roughly equivalent to runlevel 3, multi-user with networking, no GUI) and graphical.target (runlevel 5, with a display manager) are the two most common. systemctl get-default shows which target boots by default, and systemctl isolate <target> switches to another target immediately (used carefully, since this can stop a lot of running services). Sockets (.socket units) and timers (.timer units) follow the same unit-file model as services and are the modern equivalent of inetd socket activation and cron scheduling, respectively.

systemctl stop and systemctl disable are independent operations, and confusing them is a common operational mistake: systemctl disable nginx alone does NOT stop a currently running nginx — it only prevents it from starting on the NEXT boot, leaving it running right now. Conversely, systemctl stop nginx alone stops it immediately but it will start again on the next reboot if it was previously enabled. To fully take a service offline both now and permanently, you need both systemctl stop and systemctl disable (or the shortcut systemctl disable --now nginx).

Inspecting Logs with journalctl

systemd's logging component, journald, captures stdout/stderr from every service it manages plus kernel and boot messages, stored in a structured binary format queried via journalctl rather than scattered across separate plain-text log files under /var/log the way older syslog-based systems worked (though many distributions still run rsyslog alongside journald for compatibility). journalctl -u <service> filters to a specific unit's logs, -f follows in real time like tail -f, -b shows logs since the current boot, and --since/--until filter by time range — this centralized, unit-aware logging is one of the most practical day-to-day benefits of running systemd.

🏏

Cricket analogy: Like a centralized match-day log capturing every player's radio chatter and umpire decisions in one structured feed instead of scattered scorebooks, journalctl -u <service> filters to one player's contributions and -f follows commentary live.

bash

# Follow a service's logs live
journalctl -u nginx -f

# Logs since the last boot only
journalctl -u nginx -b

# Logs from the last hour
journalctl -u nginx --since "1 hour ago"

# Only errors and above
journalctl -u nginx -p err

# Kernel ring buffer messages (dmesg equivalent)
journalctl -k

systemd runs as PID 1 and manages services, sockets, timers, and mounts as declarative 'units', replacing older SysV init scripts.
systemctl start/stop control a unit right now; systemctl enable/disable control whether it launches automatically at boot — these are independent.
Custom units go in /etc/systemd/system/ and require systemctl daemon-reload after any edit before changes take effect.
Restart=on-failure (plus RestartSec) gives services automatic crash recovery without any custom supervisor script.
journalctl -u <service> -f is the standard way to tail a specific service's logs in real time.
Use systemctl disable --now <service> to both stop a service immediately AND prevent it from starting on the next boot.

Practice what you learned

1. What is the practical difference between `systemctl start nginx` and `systemctl enable nginx`?

2. After editing a custom unit file in /etc/systemd/system/, what command must be run before systemctl will recognize the changes?

3. In a .service unit file, what does `Restart=on-failure` accomplish?

4. Which command tails the live logs for a specific systemd-managed service?

5. If a currently-running, previously-enabled service is stopped with `systemctl stop myapp` but NOT disabled, what happens on the next reboot?

Was this page helpful?

Scheduling Tasks with cron

Related Study Notes in DevOps

Browse all study notes

DevOps

Nginx Study Notes

DevOps · 30 topics

DevOps

Ansible Study Notes

DevOps · 30 topics

DevOps

Advanced Kubernetes Study Notes

Kubernetes · 30 topics

DevOps

Advanced Bash Scripting Study Notes

Bash · 30 topics

DevOps

Apache Kafka Study Notes

Kafka · 30 topics

DevOps

Docker & Kubernetes Study Notes

YAML · 40 topics

Topics covered

#Bash#LinuxShellScriptingStudyNotes#DevOps#SystemdAndManagingServices#Systemd#Managing#Services#Unit#StudyNotes#SkillVeris

systemd and Managing Services