N-gram Model
An n-gram model is a probabilistic language model that predicts or represents text based on sequences of n consecutive words (or characters), estimating the probability of a word given the preceding n-1 words using frequency counts from a…
Definition
An n-gram model is a probabilistic language model that predicts or represents text based on sequences of n consecutive words (or characters), estimating the probability of a word given the preceding n-1 words using frequency counts from a training corpus.
Overview
An n-gram is simply a contiguous sequence of n items — usually words, though characters or subword units are also used — extracted from text. A 'unigram' is a single word, a 'bigram' is a sequence of two consecutive words, a 'trigram' is three consecutive words, and so on for higher values of n. N-gram language models use these sequences to estimate the probability of a word occurring given the previous n-1 words, based on how frequently that sequence appeared in a training corpus, applying the Markov assumption that only a limited window of preceding context matters for predicting the next word, rather than the entire preceding text. For example, a bigram model estimates the probability of a word based only on the single word immediately preceding it, computed as the count of that specific two-word sequence divided by the count of the first word appearing anywhere in the corpus. Higher-order n-gram models (trigrams, four-grams, and beyond) capture more context and produce more fluent predictions, but require exponentially more training data to reliably estimate probabilities for the growing number of possible sequences, and they still cannot capture long-range dependencies beyond their fixed window. N-gram models were the dominant approach to statistical language modeling for decades, powering early machine translation, speech recognition, and predictive text systems, and they require techniques like smoothing (such as Laplace or Kneser-Ney smoothing) to assign small nonzero probabilities to sequences that were never observed in the training corpus, avoiding the problem of assigning zero probability to any unseen sequence. N-gram features are also used outside of pure language modeling, particularly as a simple way to extend bag-of-words and TF-IDF text representations to partially capture local word order — for example, using both unigrams and bigrams as vocabulary items lets a model distinguish 'not good' from 'good' in a way that plain unigram bag-of-words cannot. N-gram language models have largely been superseded for generative language modeling by neural approaches, first recurrent neural networks and now transformer-based large language models, which can capture much longer and more flexible dependencies, but n-grams remain conceptually foundational and practically useful in lightweight NLP pipelines, autocomplete systems, and text feature engineering.
Key Concepts
- Models text as sequences of n consecutive words or characters
- Estimates next-word probability using frequency counts from a training corpus
- Relies on the Markov assumption that only recent context matters
- Higher-order n-grams capture more context but need exponentially more data
- Requires smoothing techniques to handle previously unseen sequences
- Historically foundational for machine translation and speech recognition
- Used as feature extensions to bag-of-words and TF-IDF representations
- Largely superseded for generative modeling by neural and transformer-based models