Why Shards Exist
An Elasticsearch index is logically one collection of documents, but physically it is split into one or more shards, each of which is a fully functional, independent Apache Lucene index. Sharding lets an index's data and indexing/query workload be distributed across multiple nodes, which is what allows Elasticsearch to hold more documents than would fit, or be searchable fast enough, on a single machine. The number of primary shards is fixed at index creation time (in modern versions it can be changed via reindexing or the split/shrink APIs, but not adjusted in place casually), so choosing shard count is a real capacity-planning decision, not a runtime toggle.
Cricket analogy: A day-night Test match splits play across multiple sessions handled by different ground staff shifts, just as an index splits its data across multiple shards each handled independently.
Primary and Replica Shards
Every primary shard can have zero or more replica shards, which are exact copies kept on different nodes. Replicas serve two purposes: high availability (if the node holding a primary shard dies, an in-sync replica is promoted to primary, so no data is lost) and read scalability (search requests can be served by any in-sync copy, primary or replica, so more replicas mean more nodes can share query load). Writes always go to the primary first, which validates and indexes the document, then forwards the operation to all replicas; a write is only acknowledged (with the default wait_for_active_shards) once it succeeds on the required number of active shard copies.
Cricket analogy: A stadium keeping a backup floodlight generator that can instantly take over if the main one fails, so play never stops, mirrors a replica shard being promoted to primary on node failure.
Choosing Shard Counts
Oversharding (too many small shards) wastes memory and cluster overhead because every shard, even an empty one, consumes file handles, memory for its Lucene segments, and adds to cluster-state size and search fan-out cost; undersharding (too few, oversized shards) makes an index hard to rebalance, slow to recover after a node failure, and risks hitting practical shard size limits (commonly recommended to keep shards in the tens of gigabytes, not hundreds). A widely used heuristic is to size shards between roughly 10GB and 50GB and to plan primary shard count for the data volume you expect at the end of the index's life, using time-based indices (daily/weekly, often automated with Index Lifecycle Management) to keep individual shards from growing unbounded.
Cricket analogy: Splitting a squad into too many tiny practice groups wastes coaching time on overhead, while cramming everyone into one giant net session makes it unmanageable — shard sizing needs the same balance.
PUT /orders-2026.07
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1
},
"mappings": {
"properties": {
"order_id": { "type": "keyword" },
"customer": { "type": "keyword" },
"amount": { "type": "double" },
"placed_at": { "type": "date" }
}
}
}
// Check shard allocation across the cluster
GET _cat/shards/orders-2026.07?v&h=index,shard,prirep,state,nodenumber_of_shards cannot be changed on an existing index without reindexing (or using the _split / _shrink APIs, which have their own constraints). Get it wrong at creation time and you either live with the consequences until the next reindex or pay the cost of a full reindex to fix it — plan shard count around projected data volume, not current volume.
Shard Allocation and Rebalancing
The master node's allocator continuously decides where each shard copy should live, respecting rules like never placing a primary and its replica on the same node, disk watermark thresholds (cluster.routing.allocation.disk.watermark.low/high/flood_stage) that stop allocating or even force read-only mode when a node's disk fills up, and optional custom rules such as allocation awareness (spreading replicas across availability zones) or shard allocation filtering (pinning certain indices to certain node attributes, e.g., hot-tier hardware). When a node leaves or joins the cluster, the allocator automatically relocates shards to restore the desired distribution, which is powerful but can generate substantial network and disk I/O, so it's tunable via cluster.routing.allocation.node_concurrent_recoveries and similar throttling settings.
Cricket analogy: A ground curator deciding which pitch each match uses based on weather and soil condition, and never scheduling two overlapping matches on the same strip, mirrors the allocator's placement rules for shards.
Use GET _cluster/allocation/explain to diagnose why a specific shard is unassigned — it reports the exact reason (e.g., disk watermark exceeded, allocation filtering rule, or no nodes matching a required attribute) rather than leaving you to guess.
- An index is split into primary shards, each an independent Lucene index, to distribute data and workload.
- Replica shards are copies of primaries used for high availability and to spread read/search load.
- Writes go to the primary first, then are forwarded to and acknowledged by replicas.
- number_of_shards is fixed at index creation and requires reindex/split to change later.
- Aim for shard sizes roughly in the tens of gigabytes; avoid both oversharding and undersharding.
- The master's allocator places shards according to rules like disk watermarks and allocation awareness.
- GET _cluster/allocation/explain diagnoses why a shard is unassigned.
Practice what you learned
1. What is a shard in Elasticsearch, physically speaking?
2. What happens when the node holding a primary shard fails and a healthy replica exists?
3. Which setting can be changed on an existing index without a full reindex?
4. What is a practical risk of oversharding an index?
5. What tool would you use to find out why a specific shard remains unassigned?
Was this page helpful?
You May Also Like
Elasticsearch Cluster Architecture
How Elasticsearch nodes join together into a cluster, elect a master, and coordinate to store and serve data reliably.
Index Lifecycle Management
How Elasticsearch's ILM feature automates moving indices through hot, warm, cold, and delete phases as data ages.
Elasticsearch Performance Tuning
Practical techniques for tuning indexing throughput, query latency, and resource usage in Elasticsearch.