Centralised logging architecture — ELK / OpenSearch stack

Every application, server, and cloud service produces logs, and in a distributed system those logs are scattered across dozens or hundreds of machines that may not even exist an hour from now. Centralised logging collects this scattered output into one searchable place, and the ELK stack, comprising Elasticsearch, Logstash, and Kibana, along with its open-source fork OpenSearch, is the most widely deployed architecture for doing exactly that at scale.

Analogy🏏Cricket

🎮 Think of it like gaming: in a large online raid every player's client records damage, deaths, and loot locally, and those machines all disconnect the instant the session ends. Reconstructing what went wrong by messaging thirty players the next day is hopeless. A central match-history server that streams every event into one queryable database as the fight unfolds is what lets you replay it perfectly afterward. Centralised logging is exactly that server for your infrastructure, capturing the fleeting output of hundreds of short-lived hosts into one durable, searchable store before the machines themselves disappear.

The architecture separates concerns across three roles. Lightweight shippers such as Filebeat or Fluentd run on every host and forward raw logs onward. Logstash, or an equivalent ingest pipeline, parses and enriches that raw text into structured fields, adding context such as source host or severity. Elasticsearch or OpenSearch then indexes the structured data for fast search, while Kibana or OpenSearch Dashboards provides the visual interface analysts actually use to query and explore it.

Analogy🏏Cricket

🎵 Think of it like music: recording an album separates clear roles — microphones on each instrument capture raw sound, the mixing desk cleans and labels every track, the master recording stores it in a searchable catalogue, and the listening room is where people finally hear it. Filebeat is the microphone shipping raw log audio, Logstash is the mixing desk parsing and enriching each line into labelled fields, Elasticsearch is the mastered catalogue indexed for instant recall, and Kibana is the room where analysts actually listen. Separating these roles is what lets each stage do its one job supremely well.

Under the hood, structured logging is what makes this architecture powerful for security work. A raw text line like a failed login buried in a wall of output is nearly useless at scale, but the same event parsed into fields such as timestamp, username, source IP, and outcome becomes instantly searchable and correlatable across every host that reported it. This structuring step is the foundation every later lesson in this module, from SIEM correlation to detection rules, depends on.

Analogy🏏Cricket

📷 Think of it like photography: a shoebox of loose, undated prints is nearly useless when you need one specific shot, but the same images tagged with date, location, camera, and subject become instantly findable in any catalogue. A raw log line buried in a wall of text is that loose print; parsing it into fields like timestamp, username, source IP, and outcome is adding the searchable tags. Once every event carries consistent metadata, you can pull every matching frame across the entire archive in seconds. That tagging step is the foundation every later search and correlation quietly depends on.

json

# Filebeat ships logs; example structured event after ingest parsing
{
  "@timestamp": "2026-07-04T02:14:09Z",
  "event.action": "login_failed",
  "user.name": "svc-deploy",
  "source.ip": "203.0.113.44",
  "host.name": "web-03",
  "log.level": "warning"
}
# Kibana query: event.action:"login_failed" AND source.ip:"203.0.113.44"

Analogy🏏Cricket

🏏 Scattered server logs are like every fielder shouting their own account of a run-out from a different corner of the ground, each in a different language. Centralised logging is the stump microphone and the big screen replay: every voice is captured, translated into the same commentary format, and shown on one screen the umpire can review instantly, instead of chasing down twelve separate memories after the fact. And just as the replay lets the third umpire rewind to the exact frame that settles the decision, a central log store lets an analyst scroll straight back to the precise event that finally explains an incident.

Best practice ships logs continuously rather than batching them hourly, so an active incident is visible in near real time rather than discovered after the fact. Enforce structured, consistent field names across every service so a single query can search all of them at once, apply retention policies that balance investigation needs against storage cost, and secure the logging pipeline itself, since attackers who can tamper with or delete logs can also erase the evidence of their own intrusion.

Analogy🏏Cricket

✈️ Think of it like travel: a well-run airline logs every bag onto a live tracking system the moment it is checked, not in a batch at midnight, uses one consistent tag format across every airport so any desk can trace a case, keeps records only as long as regulations require, and guards the tracking system itself because anyone who could delete a bag's history could hide a theft. Centralised logging follows the same discipline: stream events continuously, enforce consistent field names everywhere, set sensible retention, and secure the pipeline, since an attacker who can erase logs can erase the evidence of their own break-in.

In the real world, a security team investigating a suspicious login can query Kibana for every failed authentication from a single source IP across the entire fleet in seconds, something that would take hours of manually SSHing into individual servers and grepping through files. That speed is frequently the difference between catching an intrusion mid-attempt and only discovering it weeks later during an unrelated audit, once the damage is already done.

Analogy🏏Cricket

🎬 Think of it like movies: a detective with a searchable digital case file types one suspect's name and instantly sees every appearance across thousands of hours of footage, while the old-school investigator screens each reel by hand for days. Querying Kibana for every failed login from one source IP across the whole fleet is that instant search; SSHing into each server to grep local files is the reel-by-reel slog. The difference is not just convenience — it is often catching the intruder mid-scene rather than discovering the break-in weeks later in a reel nobody had time to watch.

Centralised logging aggregates scattered logs from every host into one searchable place.
ELK/OpenSearch separates shipping, parsing/enrichment, indexing, and visualisation into distinct roles.
Structured fields turn raw text lines into instantly searchable, correlatable events.
Ship logs continuously and enforce consistent field names across all services.
Secure the logging pipeline itself, since tampered logs can hide an intrusion's evidence.