Speculative Decoding

Definition

Speculative decoding is an inference optimization technique that speeds up large language model text generation by using a smaller, faster "draft" model to propose multiple tokens ahead, which the larger target model then verifies and accepts or rejects in a single batched pass.

Overview

Autoregressive language model inference is inherently sequential: generating each new token normally requires a full forward pass through the model, and that pass can't start until the previous token has been produced, which makes generation latency-bound even when the underlying hardware has spare compute capacity. Speculative decoding breaks this bottleneck by having a small, cheap "draft" model quickly generate a short sequence of candidate tokens, and then having the larger, more accurate "target" model verify all of those candidate tokens in a single parallel forward pass rather than one token at a time. Because verifying several tokens in one batched pass is much cheaper than generating them sequentially with the large model, and because the draft model's guesses are often correct (especially for predictable spans of text), this approach can produce the exact same output distribution as running the large model alone, just faster — the technique preserves the target model's output quality exactly, using a mathematically principled acceptance/rejection sampling scheme rather than approximating it. When the draft model's guess is wrong at some point, generation falls back to the target model's own prediction from that point forward, so the technique never produces worse output than standard decoding, only variable speedup. Speculative decoding is one of several techniques (alongside quantization, KV-cache optimization, and continuous batching) used by inference-serving systems to reduce latency and cost without changing model weights or requiring retraining. Its speedup depends heavily on how well the draft model's predictions align with the target model's — the two are sometimes distilled or trained together for better alignment — and how predictable the generated text is, with speedups often more pronounced on structured content like code than on highly creative or unpredictable text.

Key Concepts

• Uses a small, fast draft model to propose multiple tokens ahead of the target model
• Large target model verifies proposed tokens in one parallel batched pass
• Produces mathematically identical output distribution to standard decoding
• Never produces lower-quality output than running the target model alone
• Speedup depends on draft/target model alignment and text predictability
• Falls back to standard token-by-token generation when a guess is rejected
• Complements other inference optimizations like quantization and continuous batching
• Widely used in production LLM serving systems to reduce latency

Use Cases

Frequently Asked Questions

Related in AI Fundamentals