GPT-2
By OpenAI
5-billion-parameter scale and for its unusually staged public release, which OpenAI initially limited over concerns about potential misuse for generating convincing fake text.
Definition
GPT-2 is a decoder-only Transformer-based language model released by OpenAI in 2019, notable for its 1.5-billion-parameter scale and for its unusually staged public release, which OpenAI initially limited over concerns about potential misuse for generating convincing fake text.
Overview
GPT-2 followed the original GPT model and demonstrated that scaling up a decoder-only Transformer trained purely to predict the next word in text could produce a model with surprisingly broad, general-purpose language capabilities — coherent long-form text generation, rudimentary translation, and summarization — without task-specific training, simply by conditioning on a prompt. OpenAI trained GPT-2 on WebText, a dataset of roughly 8 million web pages curated by following outbound links from Reddit posts with a minimum engagement threshold, in an effort to capture higher-quality, more diverse text than a raw web crawl. GPT-2 was released in multiple sizes, with the largest (1.5B parameters) initially withheld from public release; OpenAI cited concerns about malicious uses such as generating fake news or spam at scale, a decision that sparked significant debate in the AI research community about responsible disclosure of capable models. OpenAI eventually released the full model several months later after finding limited evidence of large-scale misuse. GPT-2's "zero-shot" task performance — its ability to perform tasks it was never explicitly trained on, purely by being prompted appropriately — was an early demonstration of what would become a defining property of large LLMs: capabilities that emerge from scale and broad pretraining rather than explicit task-specific engineering. This insight directly shaped the design of OpenAI's GPT-3, which scaled the same core approach roughly 100x further and popularized prompting as the primary interface for using language models — a lineage explored further in the Large Language Models course.
Key Features
- Decoder-only Transformer architecture trained on next-token prediction
- Largest version has 1.5 billion parameters
- Trained on WebText, a curated dataset of roughly 8 million web pages
- Demonstrated zero-shot task performance without task-specific fine-tuning
- Staged public release due to initial misuse concerns
- Direct architectural predecessor to GPT-3