ELECTRA
ELECTRA is a transformer-based language model pretraining method introduced by Google Research and Stanford in 2020 that replaces masked language modeling with a more sample-efficient "replaced token detection" objective.
Definition
ELECTRA is a transformer-based language model pretraining method introduced by Google Research and Stanford in 2020 that replaces masked language modeling with a more sample-efficient "replaced token detection" objective.
Overview
ELECTRA (Efficiently Learning an Encoder that Classifies Token Replacements Accurately) was introduced in the 2020 paper "ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators." Unlike BERT, which masks roughly 15% of input tokens and trains the model to predict them, ELECTRA uses a small generator network to replace some tokens with plausible alternatives, and then trains a larger discriminator network to detect which tokens in the sequence were replaced. This "replaced token detection" objective lets the model learn from every token in the input rather than only the masked subset, making pretraining substantially more sample-efficient. Because the discriminator receives a learning signal from all input positions instead of just 15% of them, ELECTRA models can match or exceed the performance of BERT-style models while using significantly less compute during pretraining. Google demonstrated that a small ELECTRA model could outperform a much larger GPT model on the GLUE benchmark using a fraction of the compute, and that a full-scale ELECTRA-Large model achieved state-of-the-art results on GLUE and SQuAD 2.0 with less pretraining compute than comparable RoBERTa and XLNet models. ELECTRA's generator-discriminator setup is conceptually similar to a generative adversarial network (GAN), though it isn't trained adversarially in the same way — the generator is trained via standard masked language modeling rather than to fool the discriminator, which the authors found more stable to train. ELECTRA's efficiency-focused pretraining objective influenced subsequent encoder models, including DeBERTa, and it remains a popular choice for fine-tuning on classification and understanding tasks in settings where inference cost and model size matter, without sacrificing much accuracy compared to larger BERT variants.
Key Concepts
- Replaced token detection objective instead of masked language modeling
- Generator-discriminator architecture inspired loosely by GANs
- Learns from every token in the input, improving sample efficiency
- Matches or exceeds BERT-style performance with far less pretraining compute
- Released in Small, Base, and Large sizes
- State-of-the-art GLUE and SQuAD 2.0 results at release relative to compute used