Jamba
By AI21 Labs
Jamba is a family of large language models from AI21 Labs built on a hybrid architecture combining Transformer and Mamba (state space model) layers with mixture-of-experts, designed to handle very long context windows with lower memory and…
Definition
Jamba is a family of large language models from AI21 Labs built on a hybrid architecture combining Transformer and Mamba (state space model) layers with mixture-of-experts, designed to handle very long context windows with lower memory and compute overhead than pure-Transformer models.
Overview
Jamba's core innovation is architectural: rather than relying purely on the Transformer's attention mechanism, it interleaves Transformer layers with Mamba layers, a type of state space model (SSM) that processes sequences with computation and memory that scale roughly linearly with sequence length instead of the quadratic scaling that comes with standard self-attention. By combining the two, Jamba aims to retain the in-context reasoning quality Transformers are known for while gaining the long-context efficiency benefits Mamba-style architectures offer, particularly a much smaller key-value cache footprint at long context lengths. Jamba further layers in mixture-of-experts (MoE), activating only a subset of its total parameters per token, which lets AI21 scale up total model capacity without proportionally increasing inference compute cost. This combination made Jamba one of the first production-grade models to demonstrate a hybrid SSM-Transformer-MoE design at scale, supporting context windows in the hundreds of thousands of tokens with meaningfully lower memory requirements than a comparably sized pure-Transformer model would need for the same context length. AI21 Labs, an Israeli AI company with a long history in language modeling, has released Jamba both as open-weight checkpoints for research and self-hosting, and as a hosted API product, positioning it particularly for long-document analysis, extended conversational memory, and other use cases where processing very long inputs efficiently matters more than squeezing out the last few points of benchmark performance versus pure-Transformer frontier models.
Key Concepts
- Hybrid architecture combining Transformer attention layers with Mamba state-space layers
- Mixture-of-experts design activating only a subset of parameters per token
- Near-linear compute/memory scaling with sequence length versus quadratic attention
- Significantly smaller key-value cache footprint at long context lengths
- Supports very long context windows in the hundreds of thousands of tokens
- Released as both open-weight checkpoints and a hosted API by AI21 Labs
- Designed to balance Transformer reasoning quality with SSM efficiency
- One of the first production-scale hybrid SSM-Transformer-MoE models