T5
By Google Research
T5 (Text-to-Text Transfer Transformer) is an encoder-decoder Transformer model introduced by Google Research in 2019 that reframes every NLP task — translation, summarization, classification, question answering — as a single unified…
Definition
T5 (Text-to-Text Transfer Transformer) is an encoder-decoder Transformer model introduced by Google Research in 2019 that reframes every NLP task — translation, summarization, classification, question answering — as a single unified text-to-text problem, where both the input and the output are always plain text strings.
Overview
T5's central idea is deceptively simple: instead of building different model heads or output formats for different tasks, every task is expressed as "input text in, output text out." A translation task might be framed as the input string "translate English to German: That is good." with the expected output "Das ist gut."; a classification task becomes an input prompt with the class name as the output text. This unification let Google train and evaluate one architecture across a huge diversity of tasks using the same training objective and loss function. Unlike encoder-only models such as BERT, T5 uses a full encoder-decoder Transformer architecture similar to the original machine translation Transformer design — the encoder processes the input text, and the decoder generates the output text token by token, making T5 naturally suited to generative tasks like summarization and translation, not just classification. T5 was pretrained on the Colossal Clean Crawled Corpus (C4), a cleaned, deduplicated version of Common Crawl web text totaling around 750GB, using a "span corruption" objective where contiguous spans of text are masked and the model must reconstruct them. Google released T5 in multiple sizes, from Small (60M parameters) to 11B parameters (T5-11B), enabling systematic study of how performance scales with model size. T5's text-to-text framing directly influenced later instruction-tuned models and was extended in follow-up work such as mT5 (multilingual T5) and Flan-T5 (instruction-tuned T5), the latter becoming a popular open-weight base model for research into instruction following prior to the dominance of decoder-only chat models like ChatGPT. T5's encoder-decoder design and the broader shift toward general-purpose language architectures are covered in the Large Language Models course, and the Hugging Face Transformers course walks through fine-tuning models like T5 using the Transfer Learning techniques it popularized.
Key Concepts
- Encoder-decoder Transformer architecture suited to both understanding and generation
- Unified text-to-text framing for every NLP task, using one training objective
- Pretrained on the Colossal Clean Crawled Corpus (C4), ~750GB of cleaned web text
- Span corruption pretraining objective, masking contiguous text spans
- Released across multiple sizes from 60M to 11B parameters for scaling studies
- Extended into mT5 (multilingual) and Flan-T5 (instruction-tuned) variants