Capstone — Automated Server Health-Check Suite

What You'll Build

In this capstone project you will build `cricket_healthsuite.sh` — a production-grade automated health-check suite that combines every skill from the Linux & Shell Scripting course into a single cohesive tool. The suite runs a comprehensive set of checks against a CricketPulse server, produces a structured report consumable by both humans and monitoring systems, and exits with a meaningful exit code that CI/CD pipelines and alerting tools can act on.

The health-check suite covers six categories: system resources (CPU, memory, disk), process health (service running, PID valid, no zombies), filesystem integrity (FHS-correct paths, permissions, symlink validity), network connectivity (ports listening, external DNS, TLS certificate expiry), application health (HTTP endpoints responding, log files active, no recent errors), and scheduled job verification (cron jobs ran, logrotate is current).

This is not a toy script — it is the kind of tool that DevOps and SRE teams maintain in production and run as part of their deployment verification pipeline, post-incident verification checklist, and weekly infrastructure review. Building it from scratch cements all twenty-three lessons into a practical, reusable artefact that demonstrates real operational competence.

Prerequisites

• Completion of all 23 preceding lessons — this capstone exercises every concept covered across all six modules.
• A Linux server (Ubuntu 20.04+) with sudo access, the CricketPulse directory structure set up, and a running CricketPulse service managed by systemd.
• The tools `curl`, `dig`, `ss`, `openssl`, `flock`, `shellcheck` installed — install any missing with `apt-get install -y curl dnsutils openssl shellcheck`.
• Understanding that this project will take approximately 60-90 minutes to complete correctly — the final 25% is verification and edge-case handling, which takes as long as the initial 75%.
• A partner or reviewer to read through your completed script and identify any gaps in error handling or missing checks — peer review is a standard practice for production scripts.

Architecture and Design

The suite is structured as a function library with a `main` function that orchestrates the checks and a `report` function that formats output. Each check category is its own function: `check_resources`, `check_processes`, `check_filesystem`, `check_network`, `check_application`, `check_scheduled`. Each check function returns 0 on pass and non-zero on fail, accumulating failures in a global counter. The `main` function calls all checks, then generates the report and exits with the failure count.

The report format uses structured output that both humans and monitoring tools can consume. Each check line is prefixed with `[PASS]`, `[WARN]`, or `[FAIL]` — making it trivially parseable by a CI step that counts failures. The report also includes a JSON summary block at the end for monitoring systems that prefer structured data over text parsing. This dual-format output is a common pattern in production health-check tooling.

The suite uses `flock` for mutual exclusion (only one instance runs at a time), `timeout` on all network operations, `realpath` for path validation, and a cleanup trap that removes temporary files and the lock file on any exit. These hardening elements from Lesson 23 ensure the suite itself is reliable even when the system it is checking is in a degraded state — a health check tool that hangs or leaves stale locks defeats its own purpose.

Step 1 — Foundation: Framework and Resource Checks

Step 1 builds the script framework and implements `check_resources` — the CPU, memory, and disk checks. This is the module that exercises the `/proc` filesystem knowledge from Lesson 11, the threshold-based alerting logic from Lesson 16's exercise, and the structured logging and exit code patterns from Lesson 19. All subsequent steps follow the same pattern established here, so getting the framework right is essential.

The resource checks use three thresholds: WARNING at 70% (informational — monitor closely), CRITICAL at 85% (action required — investigate before next deployment), and FAILURE at 95% (immediate action — do not deploy, service may be degraded). This three-tier threshold model is more nuanced than a simple pass/fail and mirrors the alerting levels used by Prometheus, Nagios, and Datadog.

#!/bin/bash
# cricket_healthsuite.sh — Automated health-check suite
set -euo pipefail
export PS4='+(${BASH_SOURCE[0]##*/}:${LINENO}): '
[[ "${DEBUG:-}" == 'true' ]] && set -x

# === Constants ===
readonly SERVICE='cricketpulse'
readonly DEPLOY_DIR='/opt/cricketpulse'
readonly LOG_DIR='/var/log/cricketpulse'
readonly CONFIG_DIR='/etc/cricketpulse'
readonly API_PORT=8080
readonly EXTERNAL_HOST='api.cricketpulse.com'
readonly REPORT_FILE="/tmp/healthsuite_$(date +%Y%m%d_%H%M%S).txt"

# === Counters ===
TOTAL=0; PASSED=0; WARNED=0; FAILED=0

# === Lock ===
exec 9>/var/run/cricketpulse-health.lock
flock -n 9 || { echo 'Health check already running'; exit 1; }

# === Logging ===
pass()  { echo "  [PASS] $*" | tee -a "$REPORT_FILE"; ((TOTAL++, PASSED++)) || true; }
warn()  { echo "  [WARN] $*" | tee -a "$REPORT_FILE"; ((TOTAL++, WARNED++)) || true; }
fail()  { echo "  [FAIL] $*" | tee -a "$REPORT_FILE"; ((TOTAL++, FAILED++)) || true; }
section() { echo; echo "=== $1 ==" | tee -a "$REPORT_FILE"; }

# === Cleanup ===
cleanup() { local ec=$?; exit "$ec"; }
trap cleanup EXIT

# === check_resources ===
check_resources() {
    section 'SYSTEM RESOURCES'

    # Memory
    local total avail pct
    total=$(grep '^MemTotal:' /proc/meminfo | awk '{print $2}')
    avail=$(grep '^MemAvailable:' /proc/meminfo | awk '{print $2}')
    pct=$(( (total - avail) * 100 / total ))
    if   (( pct >= 95 )); then fail  "Memory: ${pct}% used — CRITICAL"
    elif (( pct >= 85 )); then warn  "Memory: ${pct}% used — HIGH"
    else                       pass  "Memory: ${pct}% used"
    fi

    # Disk — check all mounted volumes
    while IFS= read -r line; do
        local use mount
        use=$(echo "$line" | awk '{print $5}' | tr -d '%')
        mount=$(echo "$line" | awk '{print $6}')
        if   (( use >= 95 )); then fail  "Disk $mount: ${use}% — CRITICAL"
        elif (( use >= 85 )); then warn  "Disk $mount: ${use}% — HIGH"
        else                       pass  "Disk $mount: ${use}%"
        fi
    done < <(df -h --output=source,size,used,avail,pcent,target | \
             grep -v -E '^(tmpfs|devtmpfs|Filesystem)')
}

Step 2 — Process, Filesystem, and Network Checks

Step 2 implements the three middle check categories. `check_processes` uses `systemctl is-active`, `kill -0`, and `/proc` inspection from Lessons 9 and 10. `check_filesystem` applies the FHS knowledge from Lesson 3, permission checking from Lesson 7, and `find` from Lesson 6. `check_network` uses `ss`, `curl`, and `dig` from Lesson 14 with `timeout` from Lesson 23 — every network call is bounded.

The TLS certificate expiry check is particularly valuable in production suites. SSL certificate expiry is a recurring operational failure that can be eliminated entirely with automated monitoring. The check uses `openssl s_client` to retrieve the certificate and `openssl x509` to extract the expiry date, then computes the days remaining using `date` arithmetic. Warning at 30 days and failing at 7 days gives the team adequate time to renew before the certificate expires.

# Step 2: Process, filesystem, and network check functions

check_processes() {
    section 'PROCESS HEALTH'

    systemctl is-active "$SERVICE" > /dev/null 2>&1 && \
        pass "$SERVICE: active" || fail "$SERVICE: not active"

    local pid; pid=$(systemctl show "$SERVICE" --property=MainPID \
        | cut -d= -f2 2>/dev/null || echo 0)
    [[ "$pid" -gt 0 ]] && kill -0 "$pid" 2>/dev/null && \
        pass "Process running (PID $pid)" || fail 'Process not running'

    local zombies; zombies=$(ps aux | awk '$8=="Z"' | wc -l)
    (( zombies == 0 )) && pass 'No zombie processes' || \
        warn "$zombies zombie process(es) detected"
}

check_filesystem() {
    section 'FILESYSTEM INTEGRITY'

    for dir in "$DEPLOY_DIR" "$LOG_DIR" "$CONFIG_DIR"; do
        [[ -d "$dir" ]] && pass "Directory exists: $dir" || fail "Missing: $dir"
    done

    [[ -L "${DEPLOY_DIR}/current" ]] && \
        pass "Release symlink valid: $(readlink ${DEPLOY_DIR}/current)" || \
        fail 'Release symlink missing or invalid'

    local world_readable; world_readable=$(find "$CONFIG_DIR" -perm -004 -type f 2>/dev/null | wc -l)
    (( world_readable == 0 )) && pass 'No world-readable config files' || \
        fail "$world_readable world-readable file(s) in $CONFIG_DIR"
}

check_network() {
    section 'NETWORK CONNECTIVITY'

    ss -tlnp4 | grep -q ":${API_PORT} " && \
        pass "Port $API_PORT listening" || fail "Port $API_PORT not listening"

    local status; status=$(timeout 10 curl -s -o /dev/null -w '%{http_code}' \
        http://localhost:${API_PORT}/health 2>/dev/null || echo 000)
    [[ "$status" == '200' ]] && pass 'Health endpoint: 200' || \
        fail "Health endpoint: $status"

    # TLS certificate expiry
    local expiry_date days_left
    expiry_date=$(echo | timeout 10 openssl s_client \
        -connect "${EXTERNAL_HOST}:443" 2>/dev/null \
        | openssl x509 -noout -enddate 2>/dev/null \
        | cut -d= -f2) || expiry_date=''
    if [[ -n "$expiry_date" ]]; then
        local expiry_epoch now_epoch
        expiry_epoch=$(date -d "$expiry_date" +%s 2>/dev/null || echo 0)
        now_epoch=$(date +%s)
        days_left=$(( (expiry_epoch - now_epoch) / 86400 ))
        if   (( days_left <= 7  )); then fail  "TLS cert expires in ${days_left} days — CRITICAL"
        elif (( days_left <= 30 )); then warn  "TLS cert expires in ${days_left} days"
        else                             pass  "TLS cert valid for ${days_left} days"
        fi
    else
        warn 'TLS certificate check skipped (cannot reach external host)'
    fi
}