Speculative Decoding
Speculative decoding is an inference optimization technique that speeds up large language model text generation by using a smaller, faster "draft" model to propose multiple tokens ahead, which the larger target model then verifies and accepts or rejects in a single batched pass.
3 resources across 1 library
Glossary Terms(3)
Speculative Decoding
Speculative decoding is an inference optimization technique that speeds up large language model text generation by using a smaller, faster "draft" model to pro…
Knowledge Distillation
Knowledge distillation is a machine learning technique for training a smaller "student" model to reproduce the behavior of a larger, more capable "teacher" mod…
Test-Time Compute
Test-time compute refers to additional computation a language model performs at inference time — beyond a single fixed-length forward pass — such as extended r…