MPT (model)
By MosaicML (later Databricks)
MPT (MosaicML Pretrained Transformer) is a family of open-source, commercially licensed large language models released by MosaicML in 2023, notable for using ALiBi positional encoding to support very long context windows and for being…
Definition
MPT (MosaicML Pretrained Transformer) is a family of open-source, commercially licensed large language models released by MosaicML in 2023, notable for using ALiBi positional encoding to support very long context windows and for being trained with a strong focus on training efficiency.
Overview
MosaicML, a startup focused on efficient LLM training infrastructure, released MPT-7B in May 2023 as an open, commercially usable alternative to models like Meta's LLaMA (which at the time carried research-only licensing restrictions). MPT-7B was trained from scratch on 1 trillion tokens using MosaicML's own training stack (including tools later packaged as the open-source LLM Foundry and Composer libraries), and was released under the Apache 2.0 license, making it one of the first credible fully commercial open-weight 7B-parameter models. A key technical feature of MPT models is the use of ALiBi (Attention with Linear Biases) instead of standard positional embeddings, which allows the models to extrapolate to much longer context lengths at inference time than they were trained on. MosaicML demonstrated this with MPT-7B-StoryWriter-65k+, a variant fine-tuned to handle context windows of 65,000+ tokens, notably long for the time of release. The MPT family also included instruction-tuned (MPT-7B-Instruct) and chat-tuned (MPT-7B-Chat) variants, plus a larger MPT-30B released in June 2023, positioned to be trainable on a single GPU node and to outperform the original GPT-3 on several benchmarks despite being far smaller. MosaicML was acquired by Databricks in June 2023 shortly after these releases, and MPT development continued under Databricks as part of its broader open data and AI platform strategy, alongside Databricks's own later open releases such as DBRX. While MPT models are less commonly deployed today given the pace of subsequent open-weight releases (Llama 2/3, Mistral, Qwen), they were an influential early proof point that fully open, commercially licensed LLMs trained efficiently by smaller companies could compete with output from much larger labs.
Key Concepts
- Uses ALiBi positional encoding for extrapolation to very long context windows
- Released under the fully commercial Apache 2.0 license
- MPT-7B trained from scratch on 1 trillion tokens
- MPT-30B designed to be trainable on a single GPU node
- Instruction-tuned and chat-tuned variants (MPT-7B-Instruct, MPT-7B-Chat)
- MPT-7B-StoryWriter-65k+ variant supported 65,000+ token context
- Trained using MosaicML's own Composer and LLM Foundry training stack