DBRX
By Databricks
DBRX is an open-weight large language model released by Databricks (developed by its Mosaic AI research team) in March 2024, built as a fine-grained mixture-of-experts transformer with 132 billion total parameters and about 36 billion…
Definition
DBRX is an open-weight large language model released by Databricks (developed by its Mosaic AI research team) in March 2024, built as a fine-grained mixture-of-experts transformer with 132 billion total parameters and about 36 billion active per token, trained entirely on Databricks' own data and compute platform.
Overview
DBRX was released by Databricks as a demonstration of both a capable open large language model and the underlying MosaicML training platform Databricks acquired in 2023, aimed at showing enterprises that state-of-the-art model training was achievable on Databricks' own infrastructure and tooling. Architecturally, DBRX uses a fine-grained mixture-of-experts (MoE) design with 16 experts, of which 4 are activated for any given token, yielding 132 billion total parameters but only around 36 billion active parameters during inference — a design intended to deliver strong quality at meaningfully lower inference compute cost than an equivalently sized dense model. At release, Databricks reported DBRX outperforming several other openly available models of the time, including Mixtral 8x7B, Llama 2 70B, and Grok-1, on a range of standard language understanding, coding, and math benchmarks, positioning it as one of the strongest open-weight models available at that moment, while noting it trailed the very best closed frontier models like GPT-4 on several evaluations. DBRX was trained on 12 trillion tokens of curated text and code data using the full Databricks Mosaic AI training stack, intended in part as a proof point that customers could train comparably capable custom models on their own data using the same platform. Databricks released DBRX's base and instruction-tuned weights openly under a custom license permitting broad commercial use (with some restrictions for very large-scale commercial deployments), available on Hugging Face and through Databricks' own Mosaic AI Model Serving. DBRX served both as a usable open model in its own right and as a flagship marketing and technical showcase for Databricks' broader generative AI and data platform business.
Key Concepts
- Fine-grained mixture-of-experts transformer: 132B total, ~36B active parameters per token
- 16 experts with 4 activated per token, reducing inference compute relative to a dense model of similar quality
- Trained on 12 trillion tokens using Databricks' Mosaic AI training platform
- Outperformed several contemporaneous open models (Mixtral 8x7B, Llama 2 70B, Grok-1) at release
- Released with open base and instruction-tuned weights on Hugging Face
- Custom license permitting broad commercial use with restrictions for very large-scale deployments
- Available through Databricks' own Mosaic AI Model Serving platform
- Served as a technical showcase for Databricks' data and AI platform capabilities