How do you handle pagination in Elasticsearch and why is deep pagination a problem?
Learn how Elasticsearch pagination works with from/size, why deep pagination is slow, and how search_after and point-in-time solve it efficiently.
Expected Interview Answer
Elasticsearch paginates with from/size for shallow pages, but deep pagination is a problem because every shard must fetch and sort from + size documents and send them to the coordinating node, so cost grows linearly with the offset. For deep or unbounded paging you use search_after (a cursor based on the last sort values) or the scroll/point-in-time (PIT) APIs instead.
With from/size, a request for page 1000 forces each shard to build a list of from + size hits, so at from=10000 every shard sorts 10,000+ documents just to discard almost all of them — this is why index.max_result_window defaults to 10,000. search_after avoids the offset entirely by passing the sort values of the last row seen as the starting point of the next page, keeping cost constant per page. For stable, consistent deep scans you pair search_after with a point-in-time so the view of the index does not shift as documents are indexed or merged.
- from/size is simple and fine for the first few shallow pages
- search_after keeps per-page cost constant regardless of depth
- PIT gives a consistent snapshot so results do not drift mid-scan
- Avoids the memory blow-up and heap pressure of large offsets
- scroll suits one-off full exports of an entire result set
AI Mentor Explanation
Deep pagination is like a scorer who, to read out the 500th delivery of a long innings, insists on re-reading every ball from the very first one each time. Each shard is a separate scorebook forced to replay hundreds of deliveries just to reach the one page you want. search_after is instead noting 'we stopped at over 83, ball 4' and resuming from that exact marker, so you never re-count the whole innings again.
Step-by-Step Explanation
Step 1
Start with from/size
Use from and size for the first few shallow pages where the offset stays well under index.max_result_window (default 10,000).
Step 2
Add a deterministic sort
Sort by a total tiebreaker such as a unique field (or _shard_doc with PIT) so pagination order is stable and reproducible.
Step 3
Switch to search_after
For deep pages, drop from and pass the sort values of the last hit as search_after to resume exactly after it at constant cost.
Step 4
Pin a point-in-time
Open a PIT and pass its id with search_after so the index view stays frozen and results do not shift as data changes.
Step 5
Close the cursor
Close the PIT (or clear the scroll) when finished to release the retained segment resources on the cluster.
What Interviewer Expects
- Explains why from + size cost grows with the offset across all shards
- Knows index.max_result_window and its 10,000 default
- Can describe search_after with a deterministic sort key
- Understands point-in-time for consistent deep scans
- Distinguishes scroll (exports) from search_after (live paging)
Common Mistakes
- Raising max_result_window instead of switching to a cursor approach
- Using search_after without a unique tiebreaker sort field
- Using scroll for real-time user-facing pagination
- Forgetting to close PIT or scroll contexts and leaking resources
- Assuming deep from/size is only slow, not also heap-heavy
Best Answer (HR Friendly)
“Elasticsearch shows results one page at a time. Jumping far into the results with the simple method is slow because the system re-gathers everything up to that point on every request, so instead we use a bookmark-style cursor that remembers the last item seen and continues from there efficiently.”
Code Example
POST /_search
{
"size": 20,
"pit": { "id": "46ToAwMD...", "keep_alive": "2m" },
"sort": [
{ "created_at": "asc" },
{ "_shard_doc": "asc" }
],
"search_after": [ 1718000000000, 42 ]
}Follow-up Questions
- What is index.max_result_window and when would you change it?
- How does point-in-time differ from the scroll API?
- Why does search_after require a deterministic sort?
- What are the memory costs of deep from/size pagination?
- How would you paginate a UI that lets users jump to any page number?
MCQ Practice
1. Why is deep pagination with from/size expensive in Elasticsearch?
Every shard builds a from + size hit list and sends it to the coordinating node, so cost and memory grow with the offset.
2. Which approach gives constant per-page cost for deep pagination?
search_after resumes from the last hit's sort values, so it does not re-scan skipped documents and cost stays flat per page.
3. What does a point-in-time (PIT) provide during pagination?
A PIT freezes the segment view so results stay consistent even as documents are indexed or merged during a deep scan.
Flash Cards
Default max_result_window? — 10,000 — the ceiling for from + size before you must use search_after or scroll.
What is search_after? — A cursor that resumes after the last hit's sort values, giving constant per-page cost with no offset.
When to use scroll? — For one-off full exports of an entire result set, not live user-facing pagination.
Why pair search_after with PIT? — PIT freezes the index view so deep scans stay consistent and do not drift.