XLNet
XLNet is a pretraining method and transformer-based language model introduced by researchers at Google and Carnegie Mellon University in 2019, which combines autoregressive language modeling with the ability to capture bidirectional…
Definition
XLNet is a pretraining method and transformer-based language model introduced by researchers at Google and Carnegie Mellon University in 2019, which combines autoregressive language modeling with the ability to capture bidirectional context via permutation-based training.
Overview
XLNet was introduced in the 2019 paper "XLNet: Generalized Autoregressive Pretraining for Language Understanding," as a response to limitations in BERT's masked language modeling approach. BERT masks tokens during pretraining but never sees masked tokens during fine-tuning, creating a pretrain-finetune discrepancy, and it also assumes masked tokens are independent of each other given the unmasked context, which is often unrealistic. XLNet addressed both issues with a technique called permutation language modeling: instead of predicting masked tokens in a fixed left-to-right or masked pattern, it trains the model to predict tokens in a random permutation of the sequence order, allowing it to learn bidirectional context while remaining a genuinely autoregressive model. Architecturally, XLNet builds on Transformer-XL, incorporating its segment-level recurrence mechanism and relative positional encodings, which give XLNet the ability to model longer-range dependencies than the original BERT. This combination allowed XLNet to outperform BERT on a range of natural language understanding benchmarks at the time of release, including question answering, natural language inference, sentiment analysis, and document ranking tasks, achieving state-of-the-art results on 20 tasks including GLUE, SQuAD, and RACE. Despite its strong benchmark performance, XLNet saw less widespread industry adoption than BERT and its direct derivatives (like RoBERTa), partly due to its greater training complexity and computational cost. It remains an influential model in NLP research, frequently cited for its permutation-based pretraining innovation, and it informed later work on efficient and effective pretraining objectives for transformer-based language understanding models.
Key Concepts
- Permutation language modeling for bidirectional context without masking
- Built on Transformer-XL, inheriting segment-level recurrence and relative positional encodings
- Avoids the pretrain-finetune discrepancy present in BERT's masked language modeling
- Better handling of long-range dependencies than original BERT
- Achieved state-of-the-art results on GLUE, SQuAD, and RACE benchmarks at release
- Developed jointly by Google Brain and Carnegie Mellon University researchers