Reading and Managing Logs (journalctl, /var/log)

Logs are the primary evidence trail for diagnosing failures, security incidents, and performance regressions on a Linux system. Modern distributions running systemd maintain a structured, indexed, binary log store called the journal, accessed via journalctl, while many services and the older syslog ecosystem still write plain-text logs to files under /var/log. A competent operator needs to be fluent in both: the journal for kernel messages, boot sequences, and any service managed by systemd units, and /var/log flat files for applications (nginx, application servers) and historical, rotated logs. Understanding how these two systems relate — and how log rotation prevents them from consuming all available disk — is essential production knowledge.

systemd-journald collects log data from the kernel, early boot, standard output/error of services, and syslog, storing it in a structured binary format (typically under /var/log/journal for persistent storage, or /run/log/journal if persistence isn't configured). journalctl is the query interface: run alone it shows all logs in chronological order; with -u <unit> it filters to a specific systemd service; with -f it follows new entries like tail -f; with -b it shows only the current boot's logs (or -b -1 for the previous boot); and with --since/--until it filters by time range using flexible expressions like 'yesterday', '1 hour ago', or explicit timestamps. -p sets a minimum priority level (e.g., -p err shows error and above), which is invaluable for cutting through noise when triaging an incident.

Despite the journal's prevalence, /var/log remains full of important plain-text files: /var/log/syslog or /var/log/messages for general system messages (naming varies by distro family — Debian/Ubuntu use syslog, RHEL/CentOS use messages), /var/log/auth.log or /var/log/secure for authentication events including SSH logins and sudo usage, /var/log/dmesg for a snapshot of kernel boot messages, and application-specific directories like /var/log/nginx/access.log and error.log. Because these are plain text, the classic toolkit applies directly: tail -f to follow, grep to search, and less to page through with search. Many services also still log through rsyslog or syslog-ng, which can be configured to forward journal entries into these flat files, bridging the two systems.

Left unmanaged, continuously appended log files would eventually exhaust disk space. logrotate, typically run daily via a cron job or systemd timer, handles this by renaming, compressing, and eventually deleting old log files according to per-application configuration files in /etc/logrotate.d/. A typical stanza specifies rotation frequency (daily, weekly), how many rotated copies to keep, whether to compress with gzip, and postrotate scripts (commonly used to signal a service like nginx to reopen its log file handle after rotation, since the old inode has been renamed away).