Aggregation Performance

Analogy categoryAnalogies

How Aggregations Use doc_values

Aggregations read field values from doc_values, an on-disk, column-oriented structure built automatically at index time for every field except analyzed text, which makes sequential per-document access fast because it avoids the random-access pattern the inverted index is optimized for instead. Analyzed text fields don't have doc_values by default since tokenization breaks the one-value-per-document assumption aggregations rely on; if you must aggregate on text, enabling fielddata loads the field into heap memory at query time, which is discouraged in production because it's memory-hungry and can trigger circuit breaker exceptions under load.

🏏

Cricket analogy: It's like reading a scorecard's per-innings runs column straight down, sequential and fast, versus scanning ball-by-ball commentary text to reconstruct innings totals on the fly, an approach no analyst would use at scale.

Shard Count and the request_cache

Because a terms aggregation computes candidates per shard before merging results on the coordinating node, having far more shards than necessary increases coordination overhead and can worsen the accuracy problem discussed earlier in bucket aggregations, since more, smaller shards each contribute noisier top-N estimates; right-sizing shard count for the data volume is one of the most impactful aggregation performance levers. Separately, the shard-level request cache automatically caches the full response of search requests with size: 0, exactly the pattern most aggregation-only queries use, so identical aggregation requests served from cache skip re-execution entirely until the underlying index segments change.

🏏

Cricket analogy: It's like splitting a single innings' data across too many separate scorers, each with a partial view of who's top run-scorer, causing the merged summary to be noisier than a single scorer tracking the whole innings.

json

GET /logs-2026/_search
{
  "size": 0,
  "aggs": {
    "top_status_codes": {
      "terms": {
        "field": "status_code",
        "size": 10,
        "shard_size": 50
      }
    }
  }
}
// size: 0 means only aggregation results are returned,
// this exact request pattern is eligible for the shard-level
// request_cache, so repeated identical requests skip re-execution
// until the underlying segments change.

Approximate Algorithms for Scale

The cardinality aggregation trades exactness for bounded memory using HyperLogLog++, and its precision_threshold parameter is the main performance lever: lower thresholds use less memory and run faster but tolerate more estimation error on very high-cardinality fields, while raising it improves accuracy at a roughly linear memory cost. The percentiles and percentile_ranks aggregations similarly use the TDigest algorithm, which maintains a compressed sketch of the value distribution rather than sorting and storing every raw value, and its compression parameter controls the same speed/memory-versus-accuracy trade-off for tail percentiles like p99.

🏏

Cricket analogy: It's like estimating a bowling attack's economy across an entire T20 league using a quick sample of overs rather than every single ball, trading a little precision for a much faster read during a live broadcast.

The default cardinality precision_threshold of 3000 gives near-exact counts for fields with fewer than about 3000 distinct values and bounded, small error above that; each doubling of precision_threshold roughly doubles memory consumption per aggregation instance, so tune it deliberately rather than maximizing it blindly across every high-cardinality field in a dashboard.

Reducing Aggregation Cost

The single biggest lever for aggregation performance is narrowing the query context before aggregating, since aggregations only process documents that matched the query, so pushing filters into the query clause (ideally cacheable term or range filters in a filter context) shrinks the candidate set aggregations must scan. Deep aggregation trees with many nested bucket levels multiply the number of buckets combinatorially, sometimes called aggregation explosion, and for pagination through high-cardinality terms the composite aggregation is purpose-built to page through all unique combinations of one or more fields without the memory blowup a huge terms size value would cause.

🏏

Cricket analogy: It's like narrowing a stats query to just a specific series before breaking it down by innings and bowler, rather than scanning a player's entire career first, the way a focused pre-series analysis stays fast.

Requesting a terms aggregation with a very large size value, such as 100000, to try to enumerate every unique term can trigger a circuit breaker exception ('Data too large') because each shard must hold that many candidate buckets in memory before merging. Use the composite aggregation with after_key-based pagination instead when you genuinely need to walk through every unique value or combination of values in a high-cardinality field.

Aggregations read from doc_values, a fast columnar structure built at index time for every field except analyzed text.
Aggregating on analyzed text requires fielddata, which loads values into heap memory and is discouraged in production due to memory pressure.
Right-sizing shard count reduces coordination overhead and improves terms aggregation top-N accuracy.
The shard-level request_cache automatically caches size: 0 aggregation-only responses until underlying segments change.
cardinality's precision_threshold and percentiles' compression parameter both trade memory for accuracy in their respective approximate algorithms.
Filtering the query context before aggregating is the single biggest performance lever, since aggregations only scan matched documents.
The composite aggregation enables memory-safe pagination through high-cardinality terms instead of requesting an enormous terms size.