Knowledge Distillation
Knowledge distillation is a machine learning technique for training a smaller "student" model to reproduce the behavior of a larger, more capable "teacher" model, transferring much of the teacher's performance into a model that is cheaper and faster to run.
5 resources across 1 library
Glossary Terms(5)
Phi-4
Phi-4 is a small, open-weight language model from Microsoft Research, designed to deliver strong reasoning and coding performance at a fraction of the paramete…
Nemotron
Nemotron is NVIDIA's family of open large language models, optimized and distilled for efficient inference on NVIDIA GPUs, spanning general-purpose, reasoning-…
Mixture of Experts
Mixture of Experts (MoE) is a neural network architecture that replaces a single dense feed-forward layer with many smaller "expert" sub-networks and a learned…
Speculative Decoding
Speculative decoding is an inference optimization technique that speeds up large language model text generation by using a smaller, faster "draft" model to pro…
Knowledge Distillation
Knowledge distillation is a machine learning technique for training a smaller "student" model to reproduce the behavior of a larger, more capable "teacher" mod…