Top-K Sampling
Top-k sampling is a text generation strategy in which a language model restricts its choice of next token to the k highest-probability candidates at each step, then randomly samples from that reduced set according to their relative…
Definition
Top-k sampling is a text generation strategy in which a language model restricts its choice of next token to the k highest-probability candidates at each step, then randomly samples from that reduced set according to their relative probabilities.
Overview
Top-k sampling was introduced to address a key weakness of both greedy decoding, which always picks the single most probable token and often produces repetitive or generic text, and unrestricted random sampling, which draws from the model's entire probability distribution and can occasionally select extremely unlikely, low-quality, or nonsensical tokens from the distribution's long tail. Top-k sampling strikes a middle ground: at each generation step, it identifies the k tokens with the highest predicted probability, discards all other tokens from consideration entirely, renormalizes the probabilities of the remaining k tokens so they sum to one, and then samples the next token randomly according to this renormalized distribution. The parameter k controls how much randomness and diversity is introduced into generation. A small k (such as 1, which is equivalent to greedy decoding) produces highly deterministic, conservative text, while a larger k allows more of the probability distribution's tail to be considered, increasing diversity and creativity but also raising the risk of occasionally selecting a lower-quality token. Typical values of k in practice range from around 20 to 50, though the ideal value depends on the vocabulary size, the task, and the desired balance between coherence and variety. A known limitation of top-k sampling is that it uses a fixed cutoff regardless of how the model's probability distribution is actually shaped at each step: in situations where the model is very confident (probability mass concentrated on very few tokens), a fixed k may include tokens that are actually quite unlikely and inappropriate to sample; conversely, in situations where the model is very uncertain (probability spread across many plausible tokens), a fixed k may cut off legitimate, reasonable candidates. This limitation motivated the development of nucleus (top-p) sampling, which instead selects a dynamically sized set of tokens whose cumulative probability reaches a target threshold, adapting to the shape of the distribution at each step. Top-k sampling is often combined with a temperature parameter, which reshapes the probability distribution before applying the top-k cutoff, giving practitioners two complementary levers — sharpness of the distribution and breadth of the candidate pool — to tune the character of generated text.
Key Concepts
- Restricts sampling to the k most probable tokens at each generation step
- Randomly samples from the restricted, renormalized set of top candidates
- The k parameter directly controls the tradeoff between diversity and coherence
- k=1 is equivalent to greedy decoding
- Prevents sampling extremely unlikely tokens from the distribution's tail
- Uses a fixed-size cutoff regardless of the actual shape of the distribution
- Often combined with a temperature parameter for finer control
- Superseded in many applications by adaptive nucleus (top-p) sampling
Use Cases
Frequently Asked Questions
From the Blog
How LLM Temperature and Top-p Sampling Work
Temperature and top-p control how random an LLM's output is. Temperature reshapes the probability curve; top-p limits the candidate pool. Learn when to tune each.
Read More AI & TechnologyGreedy, Beam Search and Sampling: How Decoding Changes Output
Decoding is the step that turns a probability distribution into text, and it changes output more than most prompt edits do. This article compares greedy, beam search and stochastic sampling on determinism, diversity and factual drift, and names the task types each one suits.
Read More AI & TechnologySelf-Consistency Prompting: Sampling Answers and Voting
Self-consistency samples the same reasoning prompt several times at a non-zero temperature and takes the majority answer rather than trusting one chain. This covers when the technique helps, how to extract and compare answers reliably, the cost multiplier it imposes, and when a cheaper approach wins.
Read More AI & TechnologyWhy LLMs Repeat Themselves and How Sampling Settings Fix It
Language models repeat themselves because greedy and near-greedy decoding falls into self-reinforcing loops: each repeated phrase raises the probability of repeating it again. This article explains the mechanism and shows which sampling settings break the loop, which merely hide it, and which cause worse failures.
Read More