GPT-NeoX
GPT-NeoX (specifically GPT-NeoX-20B) is a 20-billion-parameter open-source autoregressive language model released by EleutherAI in 2022, along with its accompanying open-source training library of the same name.
Definition
GPT-NeoX (specifically GPT-NeoX-20B) is a 20-billion-parameter open-source autoregressive language model released by EleutherAI in 2022, along with its accompanying open-source training library of the same name.
Overview
GPT-NeoX refers both to EleutherAI's GPT-NeoX-20B model, released in February 2022, and to the GPT-NeoX training library used to build it — a codebase built on Megatron-LM and DeepSpeed for efficiently training large transformer models across many GPUs. GPT-NeoX-20B, at the time of release, was one of the largest publicly available open-source language models, following EleutherAI's earlier, smaller GPT-J-6B release. Like GPT-J, GPT-NeoX-20B was trained on the Pile dataset and uses a GPT-style decoder-only transformer architecture with rotary positional embeddings. It was trained on a cluster of GPUs provided by CoreWeave rather than the TPU infrastructure used for GPT-J, and both the trained weights and full training code were released openly, continuing EleutherAI's mission of making large-scale language model research reproducible and accessible outside major AI labs. The GPT-NeoX library itself became independently significant: it has been used well beyond EleutherAI's own models to train other notable open models, and its architecture and training-efficiency techniques influenced the broader open-source LLM training ecosystem. While GPT-NeoX-20B itself has been surpassed in capability by later open models such as Llama, Mistral, and Falcon, it remains a landmark release for demonstrating that large-scale, competitively-sized language models could be trained and shared entirely in the open, and the GPT-NeoX training framework continues to be used and maintained as infrastructure for training new models.
Key Features
- 20 billion parameters, one of the largest fully open models at release
- Decoder-only transformer with rotary positional embeddings
- Trained on the Pile dataset
- Built using the GPT-NeoX training library (based on Megatron-LM and DeepSpeed)
- Weights, code, and training details released fully open-source
- Trained on GPU infrastructure provided by CoreWeave