Elasticsearch Performance Tuning

Indexing performance is dominated by how documents arrive and how the JVM heap and disk are used to process them. Sending documents via the _bulk API in reasonably sized batches (a common starting point is 5-15MB per bulk request, tuned empirically) amortizes network and per-request overhead far better than individual index calls; increasing refresh_interval (or temporarily setting it to -1 during a large bulk load) reduces how often Lucene has to open new searchable segments, which is expensive; and setting number_of_replicas to 0 during an initial bulk load (then restoring it afterward) avoids duplicating indexing work until the data is loaded, at the cost of temporarily reduced resilience. Multiple parallel bulk indexing clients, each targeting a reasonable number of shards, generally outperform a single-threaded client no matter how large its batches are, because Elasticsearch can pipeline work across shards concurrently.

Query performance benefits from mapping and query design choices made well before the query runs: using keyword fields (not text/analyzed fields) for filters, aggregations, and sorting avoids expensive on-the-fly work; structuring queries so filter clauses live in a bool query's filter context rather than must (filter context skips scoring and is cacheable) directly cuts CPU cost; and avoiding wildcard-leading queries (e.g., *searchterm) prevents Elasticsearch from having to scan the entire term dictionary instead of using an index. The request cache (for identical, deterministic search requests with size:0, common in dashboards) and the shard-level query cache for filter contexts both help repeated queries return near-instantly, but they only help when queries are structured to be cache-friendly and deterministic (avoiding now-relative date math without rounding, for instance).

Elasticsearch's JVM heap should generally be set to no more than 50% of available RAM (leaving the rest for the OS page cache, which Lucene relies on heavily for fast segment access) and never above roughly 30-32GB, because doing so disables the JVM's compressed ordinary object pointers (compressed oops) and effectively wastes memory rather than adding capacity. Beyond heap sizing, G1GC (the default garbage collector in modern Elasticsearch/JVM combinations) generally handles Elasticsearch's allocation patterns well, but frequent long GC pauses are a strong signal of heap pressure — often from oversized aggregations, excessive field data usage, or too many shards per node — and should be diagnosed via GC logs and the node stats API rather than solved by blindly increasing heap.

Lucene stores data in immutable segments, and every indexing operation eventually triggers background merges that combine smaller segments into larger ones to keep search efficient; too many small segments slow searches down because every query has to check every segment. For read-only indices (typically after an ILM rollover), running a force merge down to a small number of segments (often 1) meaningfully reduces memory overhead and speeds up searches, since Lucene per-segment data structures (like norms and doc values) have fixed overhead regardless of segment size. This should never be done on an index still receiving writes, since a force merge is I/O-intensive and creates new segments that immediately start fragmenting again as new documents arrive.