Encoder-Decoder Model
An encoder-decoder model is a neural network architecture split into two components — an encoder that processes an input sequence into an internal representation, and a decoder that generates an output sequence from that representation —…
Definition
An encoder-decoder model is a neural network architecture split into two components — an encoder that processes an input sequence into an internal representation, and a decoder that generates an output sequence from that representation — commonly used for tasks that map one sequence to another.
Overview
The encoder-decoder pattern, also called sequence-to-sequence modeling, addresses tasks where the input and output are both sequences but may differ in length and structure, such as translating a sentence from one language to another, summarizing a long document into a short one, or converting speech audio into text. The encoder reads the entire input sequence and compresses it into an internal representation — historically a single fixed-length vector in early recurrent implementations, and later a full sequence of contextualized representations in attention-based models. The decoder then generates the output sequence step by step, conditioning each generated token on both the encoder's representation and the tokens it has already produced. Early encoder-decoder models were built from recurrent neural networks like LSTMs, but these struggled with long input sequences because compressing an entire sentence into one fixed-length vector created an information bottleneck. The introduction of the attention mechanism let the decoder look back at different parts of the encoder's output at each generation step rather than relying solely on a single compressed summary, dramatically improving performance on long sequences. This insight directly motivated the Transformer architecture, which replaced recurrence entirely with self-attention and cross-attention, becoming the standard implementation of the encoder-decoder pattern. Models like the original T5 and BART use a full encoder-decoder Transformer, well suited to tasks with clear input-output transformations, such as translation and summarization. This contrasts with decoder-only architectures like GPT, which handle generation tasks using only a single stack conditioned on all prior tokens, without a separate encoding phase, and encoder-only models like BERT, which are used for understanding tasks without any generation. The choice between these architectural patterns depends on whether a task fundamentally involves transforming one sequence into a structurally different one (favoring encoder-decoder) or open-ended generation and understanding (favoring decoder-only or encoder-only designs).
Key Concepts
- Splits the model into a separate encoder and decoder component
- Encoder compresses the input sequence into an internal representation
- Decoder generates the output sequence conditioned on the encoder's output
- Well suited to tasks mapping one sequence to a structurally different one
- Early versions built on recurrent networks, later replaced by Transformers
- Attention mechanism lets the decoder reference all encoder outputs, not just a summary
- Used in models like T5 and BART for translation and summarization
- Contrasts with decoder-only (GPT) and encoder-only (BERT) architectures