Ops Practice — Scheduled Log Rotation — Linux & Shell Scripting

What You'll Build

In this exercise you will implement a complete log management system for CricketPulse: a logrotate configuration that rotates, compresses, and retains logs on schedule, a cron job that generates a daily log summary report, and a systemd timer unit that performs weekly log archival to a separate directory. Each component is verified to work correctly before moving to the next, building a production-quality log management pipeline from first principles.

Log management is one of those operational concerns that feels like infrastructure overhead until a disk fills at 3am during a match-day traffic spike and takes the entire CricketPulse platform offline for 45 minutes. Every production team has a version of that story — and every team that implements log management properly beforehand does not. This exercise builds the habits that prevent it.

The exercise combines cron syntax, logrotate configuration, shell scripting with output redirection, systemd timer units, and journalctl verification into a single coherent system. By the end, you will have a complete, tested log management stack that handles the full lifecycle: rotation, compression, retention, reporting, and archival — all running automatically without requiring any manual intervention from the operations team.

Analogy🏏Cricket

🏏 Think of it like cricket: This script is like the pre-match ground inspection conducted by the match referee, pitch curator, and captains before a Test match begins. Before Rohit Sharma and the opposition captain walk out for the toss, the curator has measured pitch moisture, taken grass length readings, and documented the surface condition — creating a baseline against which any afternoon deterioration can be measured.Just as this structured inspection prevents surprises and creates a documented record, the inventory script creates a documented baseline for a server against which future anomalies can be compared. Just as a ground inspection without a checklist might miss a drainage issue that affects the afternoon session, a server assessment without a structured script might miss a nearly-full disk that causes a midnight deployment failure.The insight is that the value of a structured inspection is not just the current findings but the reproducible method — the same script run tomorrow highlights exactly what changed.

Prerequisites

Completion of Lessons 13–15: package management, networking basics, and cron/scheduling/logs.
A Linux system with systemd, cron, and logrotate installed (all present by default on Ubuntu 20.04+).
The CricketPulse directory structure from previous exercises with active log files — the setup commands in this lesson create sample log data if needed.
Understanding of shell arithmetic and date commands — the summary script uses these to compute log statistics.
Sudo access for installing logrotate configuration in `/etc/logrotate.d/` and systemd units in `/etc/systemd/system/`.

Setup & Project Structure

The setup creates the CricketPulse directory structure and populates the log directory with realistic sample log files of varying ages and sizes. This provides the actual log data that the log management system will operate on, making it possible to verify that rotation, compression, and retention work correctly rather than testing against an empty directory.

Analogy🏏Cricket

🏏 Think of it like cricket: building `cricket_inventory.sh` incrementally across four steps is like a batter building an innings session by session rather than swinging for six off the first ball. Just as you only need a bat and a net to start practising — no fancy equipment — this exercise needs only a text editor and a shell, nothing installed. Just as a batter grooves one shot at a time, the cover drive before the pull, each step adds exactly one report section and introduces only the commands needed to gather that piece of information. Just as marking your guard and taking centre sets a repeatable stance, creating a dedicated exercise directory and making the file executable gives you a clean, runnable base to return to. And just as a solid start makes the big total possible, getting the script structure right first means the later sections drop in without collapse. The payoff: incremental practice builds a working system-inventory script the same disciplined way a batter builds a match-winning score.

The sample log generation creates files that span the last 35 days, ensuring the 30-day retention policy has files eligible for deletion from the first test run. It also creates files that are large enough to trigger size-based rotation, allowing both time-based and size-based rotation to be tested in a single exercise session.

bash

#!/bin/bash
# setup_logs.sh — generate sample CricketPulse log data
set -euo pipefail

id cricketpulse &>/dev/null || \
    useradd --system --no-create-home --shell /usr/sbin/nologin cricketpulse

mkdir -p /var/log/cricketpulse /var/archive/cricketpulse
chown -R cricketpulse:cricketpulse /var/log/cricketpulse
chown root:cricketpulse /var/archive/cricketpulse
chmod 2775 /var/archive/cricketpulse

# Generate current app.log with realistic content
for i in $(seq 1 200); do
    echo "[$(date '+%H:%M:%S')] INFO Serving match data — request $i" \
        >> /var/log/cricketpulse/app.log
done
chown cricketpulse:cricketpulse /var/log/cricketpulse/app.log

# Generate old log files (35 days ago) — eligible for deletion
for i in 30 31 32 33 34 35; do
    OLD_FILE="/var/log/cricketpulse/app.log.$i"
    echo "Old log content from $i days ago" > "$OLD_FILE"
    touch -d "$i days ago" "$OLD_FILE"
    chown cricketpulse:cricketpulse "$OLD_FILE"
done

# Generate a large log file (10MB) to trigger size-based rotation
dd if=/dev/urandom bs=1M count=10 2>/dev/null \
    | base64 > /var/log/cricketpulse/error.log
chown cricketpulse:cricketpulse /var/log/cricketpulse/error.log

echo 'Setup complete'
ls -lh /var/log/cricketpulse/

Step 1 — Foundation: logrotate Configuration

Step 1 creates the logrotate configuration file and tests it before deployment. The configuration covers two log patterns: `app.log` with daily rotation and 30-day retention, and `error.log` with size-based rotation (rotate when it exceeds 10MB) and 7-copy retention. The critical practice is testing with `--debug` before deploying and testing the actual rotation with `--force` in a staging environment to confirm the `postrotate` signal reaches the application.

The `postrotate` script in the logrotate configuration sends `SIGUSR1` to the CricketPulse process to trigger log file reopening. Without this signal, the application continues writing to the rotated file (now named `app.log.1`) while the new `app.log` is empty — a variation of the deleted-but-open file descriptor problem from the filesystem lesson. The `|| true` at the end of the `postrotate` command prevents logrotate from failing if the service is temporarily down.

Analogy🏏Cricket

🏏 Think of it like cricket: The `postrotate` signal is like the official notification to the scorer when a new scorebook is opened mid-innings. When the first scorebook fills up and the scorer opens a fresh one, they must notify the on-field scorers to begin recording in the new book — otherwise both books might receive duplicate entries, or the old book continues receiving entries while the new one sits empty.Just as the scorer cannot simply swap books without notifying the on-field team, logrotate cannot simply rotate the log file without signalling the application to reopen its file handle. Just as the notification is sent after the new book is opened — not before — the `postrotate` script runs after the rotation has occurred, ensuring the signal reaches an application that will then open the correctly-named new file.The insight is that log rotation involves two parties — the rotation tool and the writing application — and both must coordinate for the rotation to work correctly.

bash

# /etc/logrotate.d/cricketpulse
/var/log/cricketpulse/app.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 640 cricketpulse cricketpulse
    postrotate
        systemctl kill --signal=USR1 cricketpulse 2>/dev/null || true
    endscript
}

/var/log/cricketpulse/error.log {
    size 10M
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 640 cricketpulse cricketpulse
}

# ✅ Test configuration before deploying
logrotate --debug /etc/logrotate.d/cricketpulse

# ✅ Force rotation to verify it works correctly
logrotate --force /etc/logrotate.d/cricketpulse
ls -lh /var/log/cricketpulse/
# Expected: app.log (new, empty), app.log.1 (rotated from today)