Reading and Managing Logs (journalctl, /var/log)
Logs are the primary evidence trail for diagnosing failures, security incidents, and performance regressions on a Linux system. Modern distributions running systemd maintain a structured, indexed, binary log store called the journal, accessed via journalctl, while many services and the older syslog ecosystem still write plain-text logs to files under /var/log. A competent operator needs to be fluent in both: the journal for kernel messages, boot sequences, and any service managed by systemd units, and /var/log flat files for applications (nginx, application servers) and historical, rotated logs. Understanding how these two systems relate — and how log rotation prevents them from consuming all available disk — is essential production knowledge.
Cricket analogy: Reading a match's history requires both the official scorecard (the systemd journal, structured and indexed) and the ground announcer's handwritten notes (/var/log flat files); an analyst, like a sysadmin, must be fluent in reading both sources.
The systemd Journal and journalctl
systemd-journald collects log data from the kernel, early boot, standard output/error of services, and syslog, storing it in a structured binary format (typically under /var/log/journal for persistent storage, or /run/log/journal if persistence isn't configured). journalctl is the query interface: run alone it shows all logs in chronological order; with -u <unit> it filters to a specific systemd service; with -f it follows new entries like tail -f; with -b it shows only the current boot's logs (or -b -1 for the previous boot); and with --since/--until it filters by time range using flexible expressions like 'yesterday', '1 hour ago', or explicit timestamps. -p sets a minimum priority level (e.g., -p err shows error and above), which is invaluable for cutting through noise when triaging an incident.
Cricket analogy: journalctl -u nginx is like pulling up a single bowler's spell statistics rather than the whole match; -f follows live like a running scoreboard, -b filters to the current innings, and -p err shows only wicket-taking deliveries, cutting through the noise of dot balls.
# Follow logs live, like tail -f
sudo journalctl -f
# Logs for a specific service since boot
sudo journalctl -u nginx.service -b
# Errors and above from the last hour
sudo journalctl -p err --since "1 hour ago"
# Logs for the previous boot (useful after an unexpected reboot/crash)
sudo journalctl -b -1
# Kernel ring buffer messages only
sudo journalctl -k
# Output as JSON for scripting/parsing
journalctl -u myapp.service -o json-pretty --since today
# Show disk space used by the journal, and shrink it
journalctl --disk-usage
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=2weeksBy default on many distros the journal is volatile (kept only in /run, wiped on reboot) unless /var/log/journal exists, which enables persistent storage. Create it with sudo mkdir -p /var/log/journal && sudo systemd-tmpfiles --create --prefix /var/log/journal if you need logs to survive a reboot for post-mortem analysis.
Traditional Logs in /var/log
Despite the journal's prevalence, /var/log remains full of important plain-text files: /var/log/syslog or /var/log/messages for general system messages (naming varies by distro family — Debian/Ubuntu use syslog, RHEL/CentOS use messages), /var/log/auth.log or /var/log/secure for authentication events including SSH logins and sudo usage, /var/log/dmesg for a snapshot of kernel boot messages, and application-specific directories like /var/log/nginx/access.log and error.log. Because these are plain text, the classic toolkit applies directly: tail -f to follow, grep to search, and less to page through with search. Many services also still log through rsyslog or syslog-ng, which can be configured to forward journal entries into these flat files, bridging the two systems.
Cricket analogy: /var/log/syslog (Debian) or messages (RHEL) is the ground's general commentary log, /var/log/auth.log or secure tracks who entered the pavilion (SSH/sudo), and dmesg is the pre-match pitch report; tail -f, grep, and less work on these plain-text logs like scanning a printed scorecard.
# Follow a live application log
tail -f /var/log/nginx/error.log
# Search authentication log for failed SSH attempts
sudo grep "Failed password" /var/log/auth.log
# Page through a large log file with search capability
less /var/log/syslog
# Count occurrences of an error string across rotated logs (including .gz)
zgrep -c "OutOfMemory" /var/log/myapp/*.log*
# Show the last 200 lines
tail -n 200 /var/log/syslogLog Rotation with logrotate
Left unmanaged, continuously appended log files would eventually exhaust disk space. logrotate, typically run daily via a cron job or systemd timer, handles this by renaming, compressing, and eventually deleting old log files according to per-application configuration files in /etc/logrotate.d/. A typical stanza specifies rotation frequency (daily, weekly), how many rotated copies to keep, whether to compress with gzip, and postrotate scripts (commonly used to signal a service like nginx to reopen its log file handle after rotation, since the old inode has been renamed away).
Cricket analogy: logrotate is like a groundskeeper who, on a daily schedule, rolls up old scoreboard sheets, compresses them into the archive, and signals the scoreboard operator (postrotate) to start a fresh sheet, preventing the pavilion from drowning in paperwork.
# Example /etc/logrotate.d/myapp
/var/log/myapp/*.log {
daily
rotate 14
compress
delaycompress
missingok
notifempty
create 0640 myapp myapp
postrotate
systemctl reload myapp >/dev/null 2>&1 || true
endscript
}
# Test a logrotate config without waiting for the schedule
sudo logrotate -d /etc/logrotate.d/myapp # dry run, verbose
sudo logrotate -f /etc/logrotate.d/myapp # force rotation nowIf an application keeps writing to a log file's original inode after logrotate renames it, the new file created by create will stay empty while the old, unlinked inode keeps growing invisibly — consuming disk space that du on visible files won't show. This is why postrotate hooks that signal the application to reopen its log file (e.g., systemctl reload, SIGHUP) are essential, not optional.
- journalctl queries the systemd journal; key flags are -u <unit>, -f (follow), -b (this/previous boot), --since/--until, and -p <priority>.
- The journal is persistent only if /var/log/journal exists; otherwise it is volatile and cleared on reboot.
- /var/log still holds critical plain-text logs: syslog/messages, auth.log/secure, and per-application logs under subdirectories like /var/log/nginx.
- grep, tail -f, less, and zgrep (for compressed rotated logs) are the standard toolkit for flat-file log analysis.
- logrotate manages rotation, compression, and retention via configs in /etc/logrotate.d/, preventing unbounded log growth.
- postrotate hooks must signal applications to reopen log file handles, or writes continue to a now-unlinked inode and silently waste disk space.
Practice what you learned
1. Which journalctl flag restricts output to a specific systemd service?
2. What must exist for the systemd journal to persist across reboots?
3. On Debian/Ubuntu systems, which file typically contains authentication-related events like SSH logins and sudo usage?
4. Why do logrotate configurations commonly include a postrotate block that reloads or signals the service?
5. What does `journalctl -p err --since "1 hour ago"` do?
Was this page helpful?
You May Also Like
Disk Usage and Management (df, du, mount)
Learn how to inspect filesystem capacity, measure directory sizes, and understand how block devices get attached to the Linux directory tree via mounting.
systemd and Managing Services
Learn how systemd manages services, sockets, and timers as units, and how to start, stop, enable, and inspect them with systemctl and journalctl.
grep and Regular Expressions
Master text searching with grep and the regular expression syntax that powers it, from basic literal matches to extended regex patterns and useful flags.
Monitoring Processes with ps and top
Learn to inspect running processes with ps snapshots and the interactive top monitor, reading CPU, memory, and state columns to diagnose system load.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsApache Kafka Study Notes
Kafka · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics