Mixtral 8x7B

Definition

Mixtral 8x7B is an open-weight mixture-of-experts language model released by Mistral AI in December 2023, using eight expert sub-networks per layer with only two activated per token to deliver strong performance at lower inference cost.

Overview

Mixtral 8x7B was one of the first widely adopted open-weight models to use a mixture-of-experts (MoE) architecture, in which each Transformer layer contains multiple 'expert' feed-forward sub-networks (eight, in this case) but only a small number (two) are activated for each token processed. This design let Mixtral achieve performance competitive with much larger dense models like Llama 2 70B while using significantly less compute at inference time, since not all parameters are active for every token. Mistral AI released Mixtral's weights openly, and it quickly became a popular base model for self-hosting, fine-tuning, and research into MoE architectures, contributing to renewed industry interest in mixture-of-experts designs that later appeared in models such as DeepSeek-V3 and Llama 4. Mistral AI also released an instruction-tuned variant, Mixtral 8x7B Instruct, optimized for chat and following user instructions, and the model's efficient architecture made it especially attractive for organizations wanting strong open-weight performance without the full inference cost of a comparably capable dense model.

Key Features

• Mixture-of-experts architecture: 8 experts per layer, 2 active per token
• Performance competitive with larger dense models at lower inference cost
• Openly released weights, widely used for self-hosting and fine-tuning
• Instruction-tuned Mixtral 8x7B Instruct variant for chat applications
• Helped popularize MoE architectures in the open-weight model community
• Developed by Mistral AI, released December 2023

Use Cases

Frequently Asked Questions

Related in AI Models