Self-Supervised Learning

Definition

Self-supervised learning is a machine learning technique in which a model generates its own supervisory labels from unlabeled data, learning by predicting hidden or masked parts of the input from the rest.

Overview

Self-supervised learning bridges supervised learning and unsupervised learning: it uses unlabeled data, like unsupervised learning, but creates its own training signal by constructing prediction tasks directly from that data, similar in spirit to supervised training. A classic example is masked language modeling, where random words in a sentence are hidden and the model is trained to predict them from surrounding context — this is how models like BERT are pretrained, and a close variant (predicting the next token) underlies the pretraining of most modern large language models. This approach has proven remarkably effective because it can exploit enormous quantities of freely available, unlabeled data — text scraped from the web, images without captions, audio without transcripts — to learn rich, general-purpose representations, such as embeddings that capture semantic meaning. These representations can then be adapted to specific downstream tasks through transfer learning or fine-tuning with comparatively little labeled data. Self-supervised learning is the reason today's foundation models can be trained on internet-scale data without requiring humans to manually label billions of examples, and it is a core pillar of how modern deep learning systems achieve broad general capability.

Key Concepts

• Generates its own training labels directly from unlabeled data
• Common pretext tasks include masked prediction and next-token prediction
• Enables training on massive amounts of freely available unlabeled data
• Underlies the pretraining of most modern large language models
• Produces general-purpose embeddings usable across downstream tasks
• Bridges supervised and unsupervised learning paradigms

Use Cases

Frequently Asked Questions

Related in AI Fundamentals