Perplexity (NLP)
Perplexity is a metric for evaluating language models that measures how well a model predicts a sample of text, calculated as the exponentiated average negative log-likelihood the model assigns to that text.
Definition
Perplexity is a metric for evaluating language models that measures how well a model predicts a sample of text, calculated as the exponentiated average negative log-likelihood the model assigns to that text.
Overview
Perplexity quantifies a language model's uncertainty when predicting the next token in a sequence: intuitively, it represents the weighted average number of equally likely choices the model is "confused" between at each step. A lower perplexity means the model assigns higher probability to the actual next tokens in a test sequence, indicating it models that text distribution more accurately. A perplexity of 1 would mean perfect prediction, while a very high perplexity indicates the model finds the text highly surprising or unlikely. Mathematically, perplexity is computed as two raised to the power of the cross-entropy loss (when using log base 2), or equivalently the exponential of the average negative log-likelihood per token across a text sample. It has long served as a standard intrinsic evaluation metric for language models, especially before the era of large instruction-tuned chat models, because it can be computed directly from a model's probability outputs on held-out text without requiring human judgment or a downstream task. Perplexity's main limitation is that it measures how well a model fits a particular text distribution, not how useful, safe, or aligned its generations are for real-world tasks — a model can have excellent perplexity on a benchmark corpus while still producing unhelpful or incorrect responses in conversation. It is also not directly comparable across models with different tokenizers or vocabularies, since perplexity is computed per token and tokenization schemes vary. For these reasons, modern LLM evaluation increasingly relies on task-specific benchmarks, human preference comparisons, and metrics like BLEU or ROUGE for generation quality, while perplexity remains most useful as an internal signal during pretraining to track whether a model is learning effectively.
Key Concepts
- Measures how well a language model predicts a held-out text sample
- Computed as the exponential of the average negative log-likelihood per token
- Lower perplexity indicates better predictive fit to the text distribution
- An intrinsic evaluation metric, requiring no human judgment or downstream task
- Not directly comparable across models with different tokenizers
- Widely used to track pretraining progress and model quality
- Does not directly measure helpfulness, safety, or factual accuracy
- A perplexity of 1 represents theoretically perfect prediction