Cerebras-GPT
By Cerebras Systems
Cerebras-GPT is a family of open-source GPT-style language models, ranging from 111 million to 13 billion parameters, trained by Cerebras Systems on its own wafer-scale CS-2 hardware and released to demonstrate Chinchilla-optimal,…
Definition
Cerebras-GPT is a family of open-source GPT-style language models, ranging from 111 million to 13 billion parameters, trained by Cerebras Systems on its own wafer-scale CS-2 hardware and released to demonstrate Chinchilla-optimal, compute-efficient training practices.
Overview
Released in March 2023, Cerebras-GPT served two purposes for Cerebras Systems, a company best known for building wafer-scale AI accelerator chips (the CS-2 system, built around its dinner-plate-sized Wafer Scale Engine processor): it was both an open contribution to the LLM research community and a demonstration of the company's hardware capability for training large models efficiently at scale without the complex model-parallelism engineering typically required on GPU clusters. The seven-model suite (111M, 256M, 590M, 1.3B, 2.7B, 6.7B, and 13B parameters) was trained following Chinchilla scaling laws — the DeepMind-derived finding that many earlier large models were undertrained relative to their parameter count, and that better performance per unit of compute comes from training smaller models on proportionally more tokens. Cerebras published detailed scaling-law analysis alongside the models, making Cerebras-GPT a useful reference point for researchers studying compute-optimal training, similar in spirit to how Pythia serves as a reference for scale-isolated experiments. All models were released under the Apache 2.0 license with full training details disclosed, continuing a trend (alongside EleutherAI's Pythia and, later, Ai2's OLMo) of open, reproducible LLM releases distinct from the mostly closed-data "open-weight" releases from large commercial labs. Cerebras-GPT is less commonly used directly in production applications today, given its modest scale relative to newer open models like Llama 3 or Mistral's releases, but it remains a notable historical marker both for open, compute-efficient training research and as a public proof point for Cerebras's wafer-scale hardware approach to AI training.
Key Features
- Seven model sizes from 111M to 13B parameters
- Trained entirely on Cerebras's own wafer-scale CS-2 hardware
- Follows Chinchilla-optimal, compute-efficient scaling principles
- Released under the fully permissive Apache 2.0 license
- Published alongside detailed scaling-law analysis
- Demonstrates large-model training without traditional GPU model parallelism
- Full training details and hyperparameters disclosed for reproducibility