Top-p Sampling
Top-p sampling, also called nucleus sampling, is a decoding strategy that selects the next token from the smallest set of candidates whose cumulative probability meets a specified threshold p, discarding the long tail of unlikely tokens.
Definition
Top-p sampling, also called nucleus sampling, is a decoding strategy that selects the next token from the smallest set of candidates whose cumulative probability meets a specified threshold p, discarding the long tail of unlikely tokens.
Overview
Top-p sampling was introduced as an alternative to simpler decoding strategies like top-k sampling, which always considers a fixed number of top candidates regardless of how the model's confidence is distributed. Instead, top-p dynamically adjusts the candidate pool size at each generation step: the model's next-token probabilities are sorted from highest to lowest, and tokens are added to the sampling pool until their cumulative probability reaches the threshold p (for example, 0.9). If the model is very confident, this nucleus might contain only one or two tokens; if it is uncertain, the nucleus expands to include many more. This adaptiveness is the key advantage over top-k sampling: in situations where the model has a sharply peaked distribution, top-p avoids needlessly including low-probability, potentially incoherent tokens that a fixed top-k would still allow. In situations where many tokens are plausible, top-p allows for more diversity than a rigid top-k cutoff would. The technique was proposed in the 2019 paper "The Curious Case of Neural Text Degeneration," which showed it produced more natural, less repetitive text than pure greedy or beam search decoding for open-ended generation tasks. Top-p is typically used together with temperature — temperature reshapes the overall probability distribution, and top-p then trims the tail before a token is randomly sampled from the remaining nucleus. Common defaults across LLM APIs set top-p around 0.9 to 1.0. Lowering top-p tightens the pool of candidate tokens and produces more focused, conservative output, similar in effect to lowering temperature, though the two parameters operate through different mechanisms and are sometimes tuned together for finer control over generation quality.
Key Concepts
- Dynamically sizes the candidate token pool based on cumulative probability mass
- Also known as nucleus sampling
- Adapts pool size to model confidence, unlike fixed top-k sampling
- Introduced in "The Curious Case of Neural Text Degeneration" (Holtzman et al., 2019)
- Commonly combined with temperature for fine-grained generation control
- Reduces repetitive or degenerate text compared to greedy decoding
- Typical default values range from 0.9 to 1.0
- Lower p values produce more focused, conservative output
Use Cases
Frequently Asked Questions
From the Blog
How LLM Temperature and Top-p Sampling Work
Temperature and top-p control how random an LLM's output is. Temperature reshapes the probability curve; top-p limits the candidate pool. Learn when to tune each.
Read More AI & TechnologyGreedy, Beam Search and Sampling: How Decoding Changes Output
Decoding is the step that turns a probability distribution into text, and it changes output more than most prompt edits do. This article compares greedy, beam search and stochastic sampling on determinism, diversity and factual drift, and names the task types each one suits.
Read More AI & TechnologySelf-Consistency Prompting: Sampling Answers and Voting
Self-consistency samples the same reasoning prompt several times at a non-zero temperature and takes the majority answer rather than trusting one chain. This covers when the technique helps, how to extract and compare answers reliably, the cost multiplier it imposes, and when a cheaper approach wins.
Read More AI & TechnologyWhy LLMs Repeat Themselves and How Sampling Settings Fix It
Language models repeat themselves because greedy and near-greedy decoding falls into self-reinforcing loops: each repeated phrase raises the probability of repeating it again. This article explains the mechanism and shows which sampling settings break the loop, which merely hide it, and which cause worse failures.
Read More