Mixture of Experts (MoE)

Definition

Mixture of Experts (MoE) is a neural network architecture design in which a model is composed of many specialized sub-networks ("experts"), with a learned routing mechanism ("gate" or "router") selecting only a small subset of experts to process each input, allowing the model's total parameter count to scale up substantially while keeping the computation cost per input roughly constant.

Overview

Most conventional neural networks are "dense": every parameter in the model is used to process every input. This means that increasing a model's capacity (its parameter count) directly increases the computational cost of every prediction it makes. Mixture of Experts breaks this coupling by dividing parts of the network — typically the feed-forward layers within a Transformer block — into multiple parallel "expert" sub-networks, and adding a small router network that examines each input token and decides which one or two experts (out of potentially dozens) should process it. Because only a small fraction of the total experts are activated for any given input, an MoE model can have a very large total parameter count — giving it substantial representational capacity and knowledge storage — while the actual compute used per token during training and inference stays close to that of a much smaller dense model. This distinction between total parameters and "active" parameters per token is central to how MoE models are typically described and compared; a model might have hundreds of billions of total parameters but only activate a much smaller fraction for any single token. MoE is not a new idea — the concept dates back to research from the early 1990s — but it saw a major resurgence in large language models through architectures like Google's Switch Transformer and GLaM, and it became widely known to the broader public through models such as Mistral AI's Mixtral and reported architectural choices in some of the largest frontier LLMs, where MoE is used specifically to reach very large effective capacity without a proportional increase in inference cost — a design trade-off explored in the Large Language Models course. MoE architectures introduce their own engineering challenges, including load balancing (ensuring different experts are utilized roughly evenly rather than the router collapsing onto a small subset of experts), increased memory requirements (since all experts' parameters must typically be held in memory even though only a few are used per token), and added complexity in distributed training and serving infrastructure compared to dense models.

Key Concepts

• Splits network layers into multiple specialized expert sub-networks
• Uses a learned router to activate only a small subset of experts per input token
• Decouples total parameter count from per-token computational cost
• Enables very large effective model capacity at lower inference compute than an equivalently sized dense model
• Popularized in modern LLMs via architectures like Switch Transformer, GLaM, and Mixtral
• Introduces engineering challenges around expert load balancing and memory requirements

Use Cases

Frequently Asked Questions

Related in AI Models