Core Cluster Health Signals
The starting point for monitoring any Elasticsearch cluster is GET _cluster/health, which reports the overall status (green/yellow/red), the number of nodes, and counts of active, relocating, initializing, and unassigned shards. Beyond that single snapshot, GET _cat/indices?v and GET _cat/shards?v give per-index and per-shard visibility, while GET _nodes/stats exposes deep per-node metrics including JVM heap usage, garbage collection counts and duration, thread pool queue sizes and rejections, disk usage, and indexing/search rates. Thread pool rejections in particular (visible under thread_pool.write.rejected or thread_pool.search.rejected) are an important early-warning signal: they mean the node's queue for that operation type filled up and Elasticsearch started actively refusing work rather than accepting it and falling further behind.
Cricket analogy: A quick glance at the scoreboard (green/yellow/red equivalent) tells you the match state instantly, but the detailed wagon-wheel and pitch map give the deep per-ball diagnostics, like _cluster/health versus _nodes/stats.
# Quick cluster overview
curl -s localhost:9200/_cluster/health?pretty
# Per-index shard and doc-count summary
curl -s "localhost:9200/_cat/indices?v&s=store.size:desc"
# Thread pool rejection counts across all nodes
curl -s "localhost:9200/_cat/thread_pool/write,search?v&h=node_name,name,active,queue,rejected"
# Deep per-node JVM and GC stats
curl -s "localhost:9200/_nodes/stats/jvm?filter_path=nodes.*.jvm.mem,nodes.*.jvm.gc"Indexing and Search Latency Metrics
Beyond structural health, you need to track the metrics that reflect user-facing experience: indexing latency and throughput (docs indexed per second, and time spent per operation, from indices.indexing in node/index stats), and search latency (query_time_in_millis divided by query_total gives an average, though percentiles from an APM or Kibana dashboard are far more actionable than a raw average, which hides tail latency). The slow log (index.search.slowlog and index.indexing.slowlog thresholds) is invaluable for catching the specific queries or bulk requests dragging performance down, since it logs the actual request body once it crosses a configured latency threshold, letting you identify a bad query pattern rather than just knowing 'something is slow.'
Cricket analogy: Tracking a bowler's average economy rate is useful, but the ball-by-ball log of exactly which deliveries got hit for six tells you what actually went wrong, mirroring average query time versus the slow log.
Alerting and External Monitoring Stacks
Production clusters are almost always monitored by an external system rather than relying on someone manually polling _cluster/health: the Elastic Stack's own Stack Monitoring (shipping metrics to a dedicated monitoring cluster via Metricbeat or internal collection) is the built-in option, while many teams instead scrape the Prometheus exporter (elasticsearch_exporter) or a similar integration into existing Prometheus/Grafana or Datadog stacks they already run for the rest of their infrastructure. Whichever stack is used, the alerts that matter most operationally are: cluster status leaving green, disk usage crossing watermark thresholds, JVM heap sustained above roughly 75-85%, thread pool rejections greater than zero, and unassigned shard count greater than zero for longer than a brief, expected window (e.g., during a rolling restart).
Cricket analogy: A team doesn't wait for a player to visibly collapse before intervening; sports scientists monitor heart-rate and workload thresholds continuously and flag concerning trends early, like alerting on heap-usage trends before an outage.
Alert on trends and rates, not just instantaneous values — a heap usage that spikes to 90% briefly during a GC cycle and drops back is normal, but heap sustained above 85% across multiple GC cycles, or a steadily climbing unassigned-shard count, is the pattern that actually predicts an incident.
Don't rely solely on cluster status color for alerting: a cluster can sit at green while a single node is silently running out of disk, thread pools are rejecting requests, or query latency has quietly tripled. Cluster status is necessary but not sufficient — pair it with resource and latency metrics.
- GET _cluster/health is the starting snapshot; _cat/indices, _cat/shards, and _nodes/stats give deeper detail.
- Thread pool rejections are an early-warning signal that a node's work queue is overwhelmed.
- Track indexing and search latency, favoring percentiles over raw averages to catch tail latency.
- Slow logs capture the actual request body for operations exceeding a configured latency threshold.
- Production clusters should be monitored by an external stack (Stack Monitoring, Prometheus/Grafana, or similar).
- Alert on heap usage trends, disk watermark thresholds, thread pool rejections, and unassigned shards.
- Cluster status color alone is insufficient; pair it with resource and latency metrics for real coverage.
Practice what you learned
1. What does a non-zero value in thread_pool.write.rejected indicate?
2. Why are latency percentiles generally more useful than a raw average query time?
3. What does the slow log capture that a simple latency metric does not?
4. Why is relying solely on cluster status color for alerting insufficient?
5. Which API call gives deep per-node metrics like JVM heap usage and GC counts?
Was this page helpful?
You May Also Like
Elasticsearch Cluster Architecture
How Elasticsearch nodes join together into a cluster, elect a master, and coordinate to store and serve data reliably.
Elasticsearch Performance Tuning
Practical techniques for tuning indexing throughput, query latency, and resource usage in Elasticsearch.
Shards and Replicas
How Elasticsearch splits indices into shards for scalability and duplicates them as replicas for resilience and read throughput.
Index Lifecycle Management
How Elasticsearch's ILM feature automates moving indices through hot, warm, cold, and delete phases as data ages.