What is analysis in Elasticsearch and how do analyzers, tokenizers, and filters work?
Understand Elasticsearch analysis: how analyzers, tokenizers and token filters turn text into searchable tokens, with examples and interview tips.
Expected Interview Answer
Analysis is the process Elasticsearch uses to convert text into searchable tokens, and it is performed by an analyzer built from three stages: character filters, a single tokenizer, and zero or more token filters.
When you index a text field, the analyzer first applies character filters to clean the raw string (for example stripping HTML), then the tokenizer splits the cleaned text into individual tokens such as words, and finally token filters transform those tokens by lowercasing, removing stop words, or adding stems and synonyms. The same analyzer normally runs again at search time so query text is processed identically to indexed text, which is why exact-match term queries can fail while match queries succeed. Elasticsearch ships built-in analyzers like standard and english, and you can compose custom analyzers from these building blocks in the index settings.
- Turns messy text into consistent searchable tokens
- Enables case-insensitive and language-aware search
- Character filters clean input before tokenizing
- Token filters add stemming, synonyms and stop-word removal
- Custom analyzers tailor search to your domain
AI Mentor Explanation
Analysis is like preparing a raw match commentary transcript for the record book. A character filter first wipes out crowd-noise markup, the tokenizer splits the commentary into individual balls and events, and token filters standardize each entry by lowercasing names and expanding abbreviations, so every delivery is stored in one consistent form the scorer can look up later without ambiguity.
Step-by-Step Explanation
Step 1
Start with raw text
The analyzer receives the original field value or query string as input.
Step 2
Apply character filters
Optionally clean the raw characters first, for example stripping HTML tags or mapping characters.
Step 3
Run the tokenizer
A single tokenizer splits the cleaned text into a stream of tokens, such as words on whitespace and punctuation.
Step 4
Apply token filters
Zero or more token filters transform the tokens: lowercase, remove stop words, stem, or add synonyms.
Step 5
Store or search tokens
The resulting tokens are written to the inverted index at index time and generated again for the query at search time.
What Interviewer Expects
- The three-stage pipeline: char filters, tokenizer, token filters
- Only one tokenizer but many filters per analyzer
- Understanding index-time vs search-time analysis
- Knowing built-in analyzers like standard and english
- How to compose a custom analyzer in index settings
Common Mistakes
- Confusing the tokenizer with token filters
- Thinking an analyzer can have multiple tokenizers
- Forgetting that analysis also runs at search time
- Applying an analyzer to a keyword field, which is not analyzed
- Ignoring the _analyze API when debugging tokens
Best Answer (HR Friendly)
“Analysis is how Elasticsearch breaks text into clean, searchable pieces. It cleans the raw text, splits it into words with a tokenizer, and then tidies those words with filters like lowercasing and removing common words, so search works the same whether you index or query.”
Code Example
PUT /blog
{
"settings": {
"analysis": {
"analyzer": {
"my_analyzer": {
"type": "custom",
"char_filter": ["html_strip"],
"tokenizer": "standard",
"filter": ["lowercase", "stop"]
}
}
}
}
}
// Inspect the tokens produced
GET /blog/_analyze
{
"analyzer": "my_analyzer",
"text": "<p>The Quick Brown Fox</p>"
}Follow-up Questions
- What is the difference between the standard and english analyzers?
- How do you test what tokens an analyzer produces?
- Can an analyzer have more than one tokenizer?
- What is the difference between a character filter and a token filter?
- How do search-time and index-time analyzers differ?
MCQ Practice
1. How many tokenizers can a single analyzer have?
An analyzer has exactly one tokenizer, but it can have zero or more character filters and token filters.
2. Which stage removes HTML tags from the raw input?
Character filters preprocess the raw text before tokenization, and html_strip removes HTML tags.
3. Which API shows the tokens an analyzer produces?
The _analyze API returns the token stream generated for a given analyzer and text, which is ideal for debugging.
Flash Cards
What are the three analyzer stages? — Character filters, one tokenizer, then token filters.
How many tokenizers per analyzer? — Exactly one; filters can be zero or many.
What does a token filter do? — Transforms tokens: lowercasing, stop-word removal, stemming, synonyms.
Which API debugs analysis? — The _analyze API, which returns the produced token stream.
Continue Learning
Related Interview Questions
What is the difference between a term query and a match query in Elasticsearch?
medium
What is the difference between keyword and text field types in Elasticsearch?
medium
What is a bool query in Elasticsearch and how do must, should, must_not, and filter work?
medium
What is the difference between Elasticsearch and Apache Solr?
medium