Temperature (LLM)
Temperature is a sampling parameter that controls the randomness of a language model's output by scaling the probability distribution over possible next tokens before a token is selected.
Definition
Temperature is a sampling parameter that controls the randomness of a language model's output by scaling the probability distribution over possible next tokens before a token is selected.
Overview
When a language model generates text, it produces a probability distribution over its entire vocabulary for the next token at each step. Temperature is a scalar applied to this distribution before sampling: a temperature near 0 sharpens the distribution, making the model nearly deterministic and pushing it to almost always choose the highest-probability token, while a higher temperature (typically above 1.0) flattens the distribution, giving lower-probability tokens a better chance of being selected and producing more varied, creative, or unpredictable output. Mathematically, temperature works by dividing the logits (raw model scores) by the temperature value before applying the softmax function that converts them into probabilities. A temperature of exactly 1.0 leaves the original distribution unchanged, values below 1.0 make the model more confident and repetitive, and values above 1.0 increase diversity at the risk of incoherence or factual drift. Setting temperature to 0 effectively turns generation into greedy decoding, always picking the single most likely next token. Temperature is one of several decoding parameters exposed by most LLM APIs, often used alongside top-p (nucleus) sampling and top-k sampling to further shape the sampling pool. In practice, developers tune temperature based on task: low temperatures (0 to 0.3) suit tasks requiring precision and consistency, like code generation, structured data extraction, or factual Q&A, while higher temperatures (0.7 to 1.0+) suit creative writing, brainstorming, or generating diverse variations of a response. Because temperature interacts with other sampling parameters, tuning it in isolation doesn't guarantee predictable results, and many production systems settle on a fixed low value to maximize output consistency and reduce hallucination variance.
Key Concepts
- Scales the logits of a model's next-token probability distribution before sampling
- Lower values (near 0) produce more deterministic, focused output
- Higher values (above 1.0) produce more diverse, creative, and unpredictable output
- A value of 0 is equivalent to greedy decoding
- Applied before the softmax step that converts logits into probabilities
- Commonly used alongside top-p and top-k sampling
- Exposed as a standard parameter in nearly all LLM inference APIs
- Task-dependent tuning: low for precision tasks, higher for creative tasks