Mixtral 8x22B
By Mistral AI
Mixtral 8x22B is an open-weight mixture-of-experts language model from Mistral AI composed of eight expert sub-networks of 22 billion parameters each, activating only a subset of experts per token to combine a large total parameter count…
Definition
Mixtral 8x22B is an open-weight mixture-of-experts language model from Mistral AI composed of eight expert sub-networks of 22 billion parameters each, activating only a subset of experts per token to combine a large total parameter count with efficient inference. Its routing mechanism selects a small number of experts to process each input token, giving it a large total parameter count while keeping per-token compute lower than a dense model of equivalent size, and it is released with open weights under a permissive license.
Overview
Mixtral 8x22B follows Mistral AI's earlier Mixtral 8x7B release, scaling up the mixture-of-experts approach with larger 22-billion-parameter experts while keeping the same sparse activation principle: for each input token, a routing network selects a small number of the eight available experts to process it, rather than running the entire model densely. This lets the model have a very large total parameter count while keeping the actual compute per token much lower than a dense model of equivalent total size. The routing network is trained jointly with the experts themselves, learning which expert or small combination of experts tends to handle a given type of input best, so that at inference time only a fraction of the model's total 8-times-22-billion parameter footprint is actually computed for any single token. The model was released with open weights under a permissive license, continuing Mistral AI's practice of publishing capable models for the community and enterprises to download, fine-tune, and self-host, in contrast to closed, API-only offerings from some other labs. It targets applications needing strong general reasoning, coding, and multilingual capability at a lower serving cost than a comparably capable dense model would require. Because sparse activation keeps per-token compute much lower than a dense model of equivalent total size, Mixtral 8x22B can offer capability closer to a very large dense model while requiring meaningfully less compute per request, though it still needs enough total memory to hold every expert even when only some are active for a given input. Mixtral 8x22B supports a substantial context window, making it suitable for tasks involving longer documents or extended conversations, and it performs competitively on standard benchmarks against other open and some closed models of the era, particularly on coding and math-related evaluations, according to Mistral AI's released evaluations. Within Mistral AI's lineup it sits above the dense Mistral 7B and alongside Mistral Large as a larger-scale option, distinguished specifically by its mixture-of-experts design, and among open mixture-of-experts models generally it followed the smaller Mixtral 8x7B as a scaled-up successor rather than a wholly new architecture. Deploying a mixture-of-experts model like Mixtral 8x22B efficiently requires infrastructure capable of handling the routing and memory demands of multiple experts, which differs from serving a standard dense transformer and can complicate deployment on more basic inference setups compared to a similarly-performing dense model. Organizations needing large-model-level reasoning, coding, or multilingual quality at a lower serving cost than an equivalently capable dense model have self-hosted it for those workloads, and its substantial context window has made it a common choice for long-document analysis pipelines run on private infrastructure. Mixtral 8x22B sits within Mistral AI's broader lineup alongside dense models like Mistral 7B and Mistral Large, giving developers a choice between mixture-of-experts efficiency and simpler dense architectures depending on their deployment constraints and performance requirements. Its mixture-of-experts design means it cannot simply be dropped into serving infrastructure built for a standard dense transformer without adjustment, since the routing and per-expert memory layout require different handling, which raises the operational bar relative to deploying a similarly sized dense model.
Key Features
- Mixture-of-experts architecture with eight 22-billion-parameter experts
- Sparse per-token expert activation for efficient inference
- Open weights released under a permissive license
- Substantial context window for long-document tasks
- Competitive performance on coding and math benchmarks
- Requires MoE-aware infrastructure for efficient deployment