The ELK Stack Overview
ELK is shorthand for Elasticsearch, Logstash, and Kibana — three open-source projects, all originally developed and now maintained together by Elastic, that combine into a pipeline for collecting, storing, searching, and visualizing data, most commonly logs and metrics. Elasticsearch is the storage and search engine at the core; Logstash ingests, transforms, and ships data into Elasticsearch; and Kibana provides the web UI for querying, dashboards, and visualization. The stack is often extended with Beats, lightweight data shippers, and the combination is then sometimes called the 'Elastic Stack'.
Cricket analogy: The ELK stack is like a broadcast operation: Beats/Logstash are the camera crews collecting raw footage, Elasticsearch is the archive that indexes every ball bowled, and Kibana is the studio graphics package turning that data into the on-screen scorecards fans see.
Logstash: Ingest, Transform, Ship
Logstash is a server-side data processing pipeline with three stages configured in a pipeline file: input, filter, and output. Inputs read data from sources like files, syslog, Kafka, or JDBC; filters transform the data — the grok filter is especially common for parsing unstructured log lines into structured fields using regex-like patterns; and outputs send the processed events onward, typically to Elasticsearch. Because filters run per-event and can be CPU-intensive (grok in particular), Logstash pipelines are often a throughput bottleneck compared to lightweight Beats, which is why many architectures use Beats to ship raw data and Logstash only where transformation is genuinely needed.
Cricket analogy: Logstash's grok filter parsing 'RAJ HIT FOUR OFF OVER 12.3' into structured fields (batsman, runs, over) is like a scorer converting a commentator's shouted call into a neat entry on the scorecard.
# logstash.conf
input {
file {
path => "/var/log/nginx/access.log"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "nginx-logs-%{+YYYY.MM.dd}"
}
}Beats (Filebeat, Metricbeat, Packetbeat, etc.) are lightweight, single-purpose data shippers written in Go with a much smaller footprint than Logstash. A common production pattern is Beats on edge servers shipping to Logstash for heavier transformation, or directly to Elasticsearch's ingest pipelines for lighter transformation needs.
Kibana: Visualization and Exploration
Kibana is the web-based UI layer of the stack. It lets users explore data with Discover (a searchable, filterable table view over raw documents), build visualizations (bar charts, line charts, heat maps, and geo maps) backed by Elasticsearch aggregations, and assemble those visualizations into dashboards. Kibana also hosts Dev Tools, a console for issuing raw Elasticsearch REST API calls directly, and, in the broader Elastic Stack, features like Alerting and machine-learning-based anomaly detection that operate on indexed data.
Cricket analogy: Kibana's dashboards turning raw ball-by-ball data into a run-rate graph and wagon wheel is like how a broadcaster's graphics team turns raw match data into the visuals fans see during a live telecast.
Kibana is purely a client of Elasticsearch — it has no independent data store. If the underlying Elasticsearch indices are deleted or the cluster is unreachable, Kibana dashboards will show no data even though the dashboard configuration itself is saved in a separate Kibana system index.
- ELK stands for Elasticsearch, Logstash, and Kibana, extended by Beats into the broader 'Elastic Stack'.
- Logstash ingests, filters/transforms, and outputs event data, commonly into Elasticsearch.
- The grok filter parses unstructured text (like log lines) into structured fields using pattern matching.
- Beats are lightweight, single-purpose shippers that are less resource-intensive than Logstash.
- Kibana provides Discover, Visualize, and Dashboard features backed by Elasticsearch queries and aggregations.
- Kibana Dev Tools lets you run raw Elasticsearch REST API calls from the browser.
- Kibana has no data of its own beyond dashboard configuration; all displayed data lives in Elasticsearch.
Practice what you learned
1. What does the 'L' in ELK stand for?
2. Which Logstash filter is most commonly used to parse unstructured log lines into structured fields?
3. What are Beats primarily used for?
4. What does Kibana's Dev Tools console let you do?
5. What happens to Kibana dashboards if the underlying Elasticsearch indices are deleted?
Was this page helpful?
You May Also Like
What Is Elasticsearch?
An introduction to Elasticsearch as a distributed search and analytics engine, why it exists, and the core concepts that make full-text search fast at scale.
The Elasticsearch REST API
How to interact with Elasticsearch over HTTP: indexing, retrieving, searching, and managing documents and indices via the REST API.
Indices, Documents, and Mappings
How Elasticsearch organizes data into indices and documents, and how mappings define the field types that control indexing and search behavior.