100% Free Forever
AI-Powered Learning
Industry Expert Content
Certificates & Badges
Learn At Your Own Pace
Linux & Shell Scripting
50 minbeginner

Services Practice — Manage a Daemon

What You'll Build

In this exercise you will write a production-quality systemd unit file for the CricketPulse API, deploy it using the correct systemd workflow, implement a graceful shutdown and restart script, and build a post-deployment health check that verifies the service started correctly and is operating within expected resource bounds. This combines process management, systemd configuration, and resource monitoring into a single cohesive task.

The exercise covers three distinct operational scenarios: initial deployment of a new service, zero-downtime configuration reload using `SIGHUP`, and graceful restart during a code update. Each scenario produces a script and a verification step, so by the end you have a complete service management toolkit that mirrors what DevOps engineers maintain in production.

The CricketPulse daemon in this exercise is simulated by a simple Bash script that loops and writes periodic log entries — this keeps the focus on the systemd and process management skills rather than application-specific concerns. The same unit file, deployment scripts, and health checks you build here apply directly to any real application service.

Analogy🏏Cricket
🏏 Think of it like cricket: This script is like the pre-match ground inspection conducted by the match referee, pitch curator, and captains before a Test match begins. Before Rohit Sharma and the opposition captain walk out for the toss, the curator has measured pitch moisture, taken grass length readings, and documented the surface condition — creating a baseline against which any afternoon deterioration can be measured.Just as this structured inspection prevents surprises and creates a documented record, the inventory script creates a documented baseline for a server against which future anomalies can be compared. Just as a ground inspection without a checklist might miss a drainage issue that affects the afternoon session, a server assessment without a structured script might miss a nearly-full disk that causes a midnight deployment failure.The insight is that the value of a structured inspection is not just the current findings but the reproducible method — the same script run tomorrow highlights exactly what changed.

Prerequisites

  • Completion of Lessons 09–11: process lifecycle and signals, systemd unit files and `journalctl`, and resource monitoring with `ps`, `top`, and `df`.
  • A Linux system with systemd (Ubuntu 20.04 or later) and sudo access — creating system services requires elevated privileges.
  • The CricketPulse directory structure from previous exercises, or the setup commands provided in this lesson to create it fresh.
  • Understanding of exit codes and `trap` from Module 2 — the deployment scripts use both for reliable cleanup.
  • Familiarity with `journalctl` for reading service logs — used throughout to verify each deployment step.

Setup & Project Structure

The exercise uses a simulated CricketPulse daemon — a Bash script that runs indefinitely, writes log entries on a configurable interval, and correctly handles `SIGTERM` for graceful shutdown and `SIGHUP` for configuration reload. This simulation is realistic enough to exercise all the systemd and process management concepts without requiring a real application to be compiled and installed.

Analogy🏏Cricket
🏏 Think of it like cricket: building `cricket_inventory.sh` incrementally across four steps is like a batter building an innings session by session rather than swinging for six off the first ball. Just as you only need a bat and a net to start practising — no fancy equipment — this exercise needs only a text editor and a shell, nothing installed. Just as a batter grooves one shot at a time, the cover drive before the pull, each step adds exactly one report section and introduces only the commands needed to gather that piece of information. Just as marking your guard and taking centre sets a repeatable stance, creating a dedicated exercise directory and making the file executable gives you a clean, runnable base to return to. And just as a solid start makes the big total possible, getting the script structure right first means the later sections drop in without collapse. The payoff: incremental practice builds a working system-inventory script the same disciplined way a batter builds a match-winning score.

Run the setup script as root to create the directory structure, service user, and daemon script. The setup also creates an intentionally broken version of the daemon that exits with code 1 on startup — used in Step 3 to verify that `Restart=on-failure` and the circuit breaker work correctly. Always verify the setup completed without errors before starting the main exercise steps.

bash
#!/bin/bash
# setup_daemon.sh — creates CricketPulse daemon simulation
set -euo pipefail

id cricketpulse &>/dev/null || \
    useradd --system --no-create-home --shell /usr/sbin/nologin cricketpulse

mkdir -p /opt/cricketpulse/bin /etc/cricketpulse /var/log/cricketpulse
chown -R cricketpulse:cricketpulse /opt/cricketpulse /var/log/cricketpulse
chown root:cricketpulse /etc/cricketpulse
chmod 750 /etc/cricketpulse /var/log/cricketpulse

# The daemon: handles SIGTERM and SIGHUP correctly
cat > /opt/cricketpulse/bin/cricketpulse-api.sh << 'EOF'
#!/bin/bash
LOG=/var/log/cricketpulse/app.log
CONFIG=/etc/cricketpulse/config.yaml
RUNNING=true
INTERVAL=5

log() { echo "[$(date '+%H:%M:%S')] $*" >> "$LOG"; }

reload_config() {
    log 'SIGHUP received — reloading configuration'
    INTERVAL=$(grep 'interval:' "$CONFIG" 2>/dev/null | awk '{print $2}' || echo 5)
    log "New interval: ${INTERVAL}s"
}

shutdown() { log 'SIGTERM received — shutting down gracefully'; RUNNING=false; }

trap reload_config SIGHUP
trap shutdown SIGTERM

log 'CricketPulse API started'
while $RUNNING; do
    log 'Serving match data...'
    sleep "$INTERVAL" &
    wait $!
done
log 'Shutdown complete'
EOF
chmod 755 /opt/cricketpulse/bin/cricketpulse-api.sh

# Default config
echo 'interval: 5' > /etc/cricketpulse/config.yaml
chown root:cricketpulse /etc/cricketpulse/config.yaml
chmod 640 /etc/cricketpulse/config.yaml
echo 'Setup complete'

Step 1 — Foundation: Write the Unit File

Step 1 is writing the systemd unit file with all the production-quality directives covered in Lesson 10. The unit file must specify the service user, working directory, start command, reload command, restart policy with circuit breaker, security hardening directives, and environment file. Each directive serves a specific purpose — writing them with understanding rather than copying them blindly is the goal of this step.

After writing the unit file, install it with the correct daemon-reload sequence and verify that systemd parsed it without errors using `systemctl status cricketpulse` before attempting to start the service. A unit file with a syntax error will cause systemd to report a 'bad unit file' error — catching this before the first start attempt saves the confusion of a service that appears to start but immediately fails.

Analogy🏏Cricket
🏏 Think of it like cricket: Writing the unit file before starting the service is like the ICC match referee verifying all playing conditions documents are correctly completed before the toss. The toss cannot happen until the conditions are formally accepted — any error in the playing conditions document must be resolved first. A unit file syntax error is exactly this: systemd refuses to use a malformed unit file, so catching the error before the first start attempt avoids the confusion of a service that appears to start but fails immediately with a cryptic error.Just as the match referee checks each section of the conditions document — pitch preparation, ball change rules, DRS availability — you should verify each section of the unit file: [Unit] dependencies, [Service] execution parameters, and [Install] boot target. The `systemctl status` check after `daemon-reload` confirms systemd has accepted the unit file without syntax errors before the service is started.The insight is that verification before execution is always cheaper than debugging after a failed start.
bash
# /etc/systemd/system/cricketpulse.service
[Unit]
Description=CricketPulse Live Scores API
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=cricketpulse
Group=cricketpulse
WorkingDirectory=/opt/cricketpulse
ExecStart=/opt/cricketpulse/bin/cricketpulse-api.sh
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/kill -TERM $MAINPID

Restart=on-failure
RestartSec=5
StartLimitIntervalSec=60
StartLimitBurst=3

StandardOutput=append:/var/log/cricketpulse/app.log
StandardError=append:/var/log/cricketpulse/error.log

UMask=0027
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/cricketpulse

[Install]
WantedBy=multi-user.target

# Deploy sequence
sudo systemctl daemon-reload
sudo systemctl status cricketpulse   # Verify no parse errors before starting
sudo systemctl enable cricketpulse
sudo systemctl start cricketpulse
sudo systemctl status cricketpulse   # Verify active (running)

Step 2 — Core Logic: Health Check and Reload

Step 2 builds a post-deployment health check script and demonstrates zero-downtime configuration reload. The health check verifies four conditions: the service is in `active (running)` state, the process is actually running (not just reported as running by systemd), the log file has recent entries (the daemon is actively working, not stuck), and no error-level journal entries occurred in the last 30 seconds.

The configuration reload demonstrates `SIGHUP` in action. Change the `interval:` value in `/etc/cricketpulse/config.yaml` and run `systemctl reload cricketpulse`. The daemon should log a reload event and switch to the new interval — all without restarting, without any gap in service, and with the PID unchanged. Verify this by confirming the PID before and after the reload is identical.

Analogy🏏Cricket
🏏 Think of it like cricket: The health check is like the umpires' check at the start of each session in a Test match — confirming all players are on the field, the ball is correctly prepared, and the fielding team is set in legal positions before calling play. Each check is binary: either the session can start or it cannot.Just as an umpire who only checks that players are present but not that the ball is correctly prepared creates a potential dispute mid-over, a health check that only verifies `systemctl is-active` but not that the process is actually responding creates false confidence. The four-condition check — service state, process running, recent log activity, no errors — provides genuine assurance that the service is working, not just that systemd believes it is running.The insight is that a health check is only as valuable as the conditions it verifies — checking that the service is 'running' without verifying it is 'working' catches less than half the failures.
bash
#!/bin/bash
# cricket_health_check.sh — post-deployment service verification
set -euo pipefail

SERVICE='cricketpulse'
FAILURES=0
pass() { echo "  [PASS] $1"; }
fail() { echo "  [FAIL] $1"; ((FAILURES++)) || true; }

echo "=== Health Check: $SERVICE ==="

# 1. systemd reports active
systemctl is-active "$SERVICE" > /dev/null && pass 'Service is active' || fail 'Service not active'

# 2. Process is actually running
PID=$(systemctl show "$SERVICE" --property=MainPID | cut -d= -f2)
kill -0 "$PID" 2>/dev/null && pass "Process running (PID $PID)" || fail 'Process not running'

# 3. Log has recent entries (active within last 30s)
LAST_LOG=$(stat -c '%Y' /var/log/cricketpulse/app.log 2>/dev/null || echo 0)
NOW=$(date +%s)
if [ $((NOW - LAST_LOG)) -lt 30 ]; then
    pass 'Log file updated in last 30s'
else
    fail 'Log file stale — daemon may be stuck'
fi

# 4. No errors in journal since startup
ERR_COUNT=$(journalctl -u "$SERVICE" -p err --since '30 seconds ago' | grep -c . || true)
[ "$ERR_COUNT" -eq 0 ] && pass 'No errors in journal' || fail "$ERR_COUNT error(s) in journal"

echo
[ "$FAILURES" -eq 0 ] && echo 'HEALTH CHECK: PASSED' && exit 0 || echo "HEALTH CHECK: FAILED ($FAILURES checks)"
exit "$FAILURES"

Step 3 — Integration: Restart and Circuit Breaker

Step 3 tests the restart policy and circuit breaker by temporarily replacing the daemon script with one that exits immediately with code 1. Trigger a restart and observe systemd attempting to restart the service, respecting the `RestartSec=5` delay between attempts. After three failures within 60 seconds (matching `StartLimitBurst=3`), systemd should enter the failed state and stop retrying.

After observing the circuit breaker activate, restore the working daemon, reset the failed state with `systemctl reset-failed`, and restart successfully. This complete cycle — failure, circuit break, fix, reset, restart — is the exact sequence that occurs during real production incidents where a misconfiguration causes a crash loop. Experiencing it in a controlled environment prevents confusion when it occurs under pressure.

Analogy🏏Cricket
🏏 Think of it like cricket: The circuit breaker is like a weather interruption protocol that stops play after a threshold of interruptions in a session. If play is interrupted for bad light three times in one hour, the umpires suspend the session entirely rather than allowing the pattern to continue. The reset is like the start of a new session — the clock resets and the pattern can begin again.Just as the interruption protocol protects players and the match from the cumulative cost of repeated short stoppages, the circuit breaker protects the server from the cumulative cost of rapid restart cycles consuming CPU and creating log noise. Just as the match must officially resume from a known state — players at designated positions, ball count confirmed — `systemctl reset-failed` provides the clean slate that allows a confident restart after the underlying problem has been fixed.The insight is that circuit breakers require an explicit reset by a human — they do not automatically clear, because automatic clearing would defeat their purpose of forcing human investigation.
bash
# Step 3: Test restart policy and circuit breaker

# Save working daemon
cp /opt/cricketpulse/bin/cricketpulse-api.sh /tmp/cricketpulse-api.sh.bak

# Replace with failing version
cat > /opt/cricketpulse/bin/cricketpulse-api.sh << 'EOF'
#!/bin/bash
echo 'Simulated startup failure' >> /var/log/cricketpulse/app.log
exit 1
EOF
chmod 755 /opt/cricketpulse/bin/cricketpulse-api.sh

# Trigger restart — watch systemd retry with 5s delay
systemctl restart cricketpulse
watch -n 1 'systemctl status cricketpulse | head -15'
# Observe: service restarts 3 times then enters 'failed' state

# Check journal for restart events
journalctl -u cricketpulse --since '2 minutes ago'

# Restore working daemon and reset failed state
cp /tmp/cricketpulse-api.sh.bak /opt/cricketpulse/bin/cricketpulse-api.sh
systemctl reset-failed cricketpulse
systemctl start cricketpulse
systemctl status cricketpulse   # Should show active (running)

# Run health check to confirm recovery
bash cricket_health_check.sh

Step 4 — Testing & Verification

Run the complete deployment, health check, reload, and restart cycle end-to-end. Verify that the health check exits with code 0 after a successful deployment, non-zero after the simulated failure, and code 0 again after recovery. Check that the PID remains unchanged through a reload operation, confirming zero-downtime configuration change. Check the journal to confirm graceful shutdown messages appear on `systemctl stop`.

Analogy🏏Cricket
🏏 Think of it like cricket: running the full script and cross-checking its numbers against `nproc`, `free -h`, and `df -h` is like a scorer confirming the big-screen total against the umpire's notebook and the third umpire's feed before it goes in the record. Just as a total that only appears on the scoreboard is untrusted until verified against an independent source, the report's CPU, memory, and disk figures are only trustworthy once they match the raw commands. Just as the score must show correctly to both the crowd in the ground and in the official book, the output should appear simultaneously in the terminal and in the timestamped file. Just as a good scorer confirms every figure comes from the right source rather than a stale note, you confirm the script reads from the correct system sources. The payoff: cross-checking against independent commands proves the inventory report is accurate, not just plausible-looking.

The final verification step runs the resource monitoring commands from Lesson 11 against the running service. Confirm that `ps aux` shows the daemon running as the `cricketpulse` user, that memory usage is stable (not growing, which would indicate a memory leak), and that the log file is being written to at the expected interval. These checks form the baseline from which future anomalies will be detectable.

bash
#!/bin/bash
# Full verification cycle
set -euo pipefail

SERVICE='cricketpulse'

echo '=== DEPLOYMENT VERIFICATION ==='

# 1. Service state
systemctl is-active "$SERVICE" && echo '[PASS] Service active' || { echo '[FAIL] Service not active'; exit 1; }

# 2. Reload: PID must not change
PID_BEFORE=$(systemctl show "$SERVICE" --property=MainPID | cut -d= -f2)
echo 'interval: 3' | sudo tee /etc/cricketpulse/config.yaml > /dev/null
systemctl reload "$SERVICE"
sleep 2
PID_AFTER=$(systemctl show "$SERVICE" --property=MainPID | cut -d= -f2)
[ "$PID_BEFORE" = "$PID_AFTER" ] && echo '[PASS] Reload: PID unchanged' || echo '[FAIL] Reload changed PID'

# 3. Graceful stop: check for shutdown log message
systemctl stop "$SERVICE"
grep 'Shutdown complete' /var/log/cricketpulse/app.log && \
    echo '[PASS] Graceful shutdown confirmed' || echo '[FAIL] No shutdown message'

# 4. Restart and resource check
echo 'interval: 5' | sudo tee /etc/cricketpulse/config.yaml > /dev/null
systemctl start "$SERVICE"
sleep 3
echo '--- Process running as correct user ---'
ps aux | grep '[c]ricketpulse-api' | awk '{print "User:", $1, "PID:", $2, "MEM:", $4"%"}'

# 5. Full health check
bash cricket_health_check.sh

Warning: Never edit a systemd unit file and run `systemctl restart` without `daemon-reload` in between. systemd caches unit files at load time — the restart will use the old configuration, not the new one. The change will appear to have no effect, leading to wasted debugging time. The sequence is always: edit → `daemon-reload` → `restart`. If you see `Warning: The unit file ... has been changed on disk. Run 'systemctl daemon-reload'` in `systemctl status` output, this is systemd telling you exactly this — the unit file has changed but systemd has not yet loaded the new version.

Extension Challenge: Extend the exercise with three additional tasks. First, add a `ExecStartPre=` directive that runs a configuration validation script before the daemon starts — if validation fails (non-zero exit), systemd will not start the service. Second, configure the journal retention for this service by adding `[Service]` directive `LogRateLimitIntervalSec=30` and `LogRateLimitBurst=1000` to prevent log flooding. Third, write a deployment script that combines all steps — stop, update binary, daemon-reload, start, health check — into a single atomic workflow that rolls back automatically if the health check fails.

  • The unit file deployment sequence is always: write file → `daemon-reload` → `enable` → `start` — skipping `daemon-reload` causes systemd to use the cached old configuration.
  • Use `ExecReload=/bin/kill -HUP $MAINPID` to enable `systemctl reload` for zero-downtime configuration changes — without this directive, `systemctl reload` has no effect.
  • The circuit breaker (`StartLimitBurst=3` within `StartLimitIntervalSec=60`) stops a crash-looping service and requires `systemctl reset-failed` after fixing the underlying problem before restart is possible.
  • A health check that only verifies `systemctl is-active` misses half the failures — also verify the process is alive with `kill -0`, the log is being written to, and no journal errors occurred.
  • The `StandardOutput=append:` and `StandardError=append:` directives capture service output to specific log files without requiring the application to implement its own logging — useful for services that only write to stdout.
  • Observe the complete failure-circuit_break-fix-reset-restart cycle in a controlled environment to avoid confusion when it occurs under production pressure for the first time.
Lesson 12 of 40
0% complete