The Analysis Pipeline
An analyzer is a pipeline of three stages applied to text fields at both index time and, by default, at query time: zero or more character filters that preprocess raw text, exactly one tokenizer that splits the text into individual tokens, and zero or more token filters that transform those tokens, such as lowercasing, removing stopwords, or stemming. Understanding this pipeline is essential because it explains why a search for 'running' can match a document containing 'runs' when a stemmer is in play.
Cricket analogy: A ball goes through the bowler's grip adjustment, the delivery action, and finally the swing off the seam, three distinct stages, much like text passes through character filters, a tokenizer, and token filters in sequence.
Character Filters and Tokenizers
Character filters operate on the raw string before tokenization, for example html_strip removes markup tags like <b> so they don't pollute indexed terms, or a mapping char filter can convert characters like ':)' into a word like 'happy'. The tokenizer then does the actual splitting; the default standard tokenizer splits on word boundaries following Unicode text segmentation rules, while alternatives like the ngram tokenizer produce overlapping substrings useful for autocomplete-style partial matching.
Cricket analogy: Stripping ground-level crowd noise from a stump-mic feed before analyzing the sound of bat on ball is like a character filter cleaning raw text before tokenization even begins.
PUT /articles
{
"settings": {
"analysis": {
"analyzer": {
"my_custom_analyzer": {
"type": "custom",
"char_filter": ["html_strip"],
"tokenizer": "standard",
"filter": ["lowercase", "stop", "snowball"]
}
}
}
},
"mappings": {
"properties": {
"body": { "type": "text", "analyzer": "my_custom_analyzer" }
}
}
}
Token Filters and the _analyze API
Token filters run last and transform the token stream, common ones being lowercase for case-insensitive matching, stop to remove common words like 'the' and 'is' that add noise without meaning, and snowball or porter_stem to reduce words to a root form so 'jumping' and 'jumped' both match 'jump'. You can test any analyzer directly against sample text using the _analyze API without indexing anything, which is invaluable for debugging why a query isn't matching as expected.
Cricket analogy: A coach running a net session purely to test a bowler's yorker before it's used in a real match is like calling _analyze to test an analyzer's output before it touches real documents.
Call GET /articles/_analyze with { "analyzer": "my_custom_analyzer", "text": "The <b>Quick</b> Foxes were Jumping!" } to see exactly which tokens the pipeline produces, this is the fastest way to debug unexpected search misses.
Changing an analyzer's configuration does not retroactively re-tokenize already-indexed documents. Existing documents keep whatever tokens were produced at their original index time, so you must reindex after changing an analyzer for the new behavior to apply consistently.
- An analyzer is a pipeline of character filters, exactly one tokenizer, and token filters, applied in that order.
- Character filters clean raw text before splitting, e.g. html_strip removes markup tags.
- The tokenizer performs the actual splitting into tokens; standard splits on Unicode word boundaries.
- Token filters transform the token stream, e.g. lowercase, stop, and stemmers like snowball.
- The _analyze API lets you preview an analyzer's output on sample text without indexing anything.
- Custom analyzers are defined in an index's settings.analysis block and referenced by name in mappings.
- Changing an analyzer does not retroactively affect already-indexed documents; a reindex is required.
Practice what you learned
1. In what order does the analysis pipeline run?
2. What does the html_strip character filter do?
3. Which API lets you preview how an analyzer will tokenize sample text without indexing anything?
4. What happens to already-indexed documents when you change an index's analyzer configuration?
5. What is the purpose of a stemmer token filter like snowball?
Was this page helpful?
You May Also Like
Mapping Types Explained
Understand the core Elasticsearch field data types, text vs keyword, numbers, dates, and objects, and how each shapes search behavior.
Dynamic vs Explicit Mapping
Compare Elasticsearch's automatic field-type inference against explicitly defined mappings, and know when to use each.
Creating an Index
Learn how Elasticsearch indices are created, configured with shards and replicas, and named for reliable, scalable search.