Self-Supervised Learning
Self-supervised learning is a machine learning technique in which a model generates its own supervisory labels from unlabeled data, learning by predicting hidden or masked parts of the input from the rest.
Definition
Self-supervised learning is a machine learning technique in which a model generates its own supervisory labels from unlabeled data, learning by predicting hidden or masked parts of the input from the rest.
Overview
Self-supervised learning bridges supervised learning and unsupervised learning: it uses unlabeled data, like unsupervised learning, but creates its own training signal by constructing prediction tasks directly from that data, similar in spirit to supervised training. A classic example is masked language modeling, where random words in a sentence are hidden and the model is trained to predict them from surrounding context — this is how models like BERT are pretrained, and a close variant (predicting the next token) underlies the pretraining of most modern large language models. This approach has proven remarkably effective because it can exploit enormous quantities of freely available, unlabeled data — text scraped from the web, images without captions, audio without transcripts — to learn rich, general-purpose representations, such as embeddings that capture semantic meaning. These representations can then be adapted to specific downstream tasks through transfer learning or fine-tuning with comparatively little labeled data. Self-supervised learning is the reason today's foundation models can be trained on internet-scale data without requiring humans to manually label billions of examples, and it is a core pillar of how modern deep learning systems achieve broad general capability.
Key Concepts
- Generates its own training labels directly from unlabeled data
- Common pretext tasks include masked prediction and next-token prediction
- Enables training on massive amounts of freely available unlabeled data
- Underlies the pretraining of most modern large language models
- Produces general-purpose embeddings usable across downstream tasks
- Bridges supervised and unsupervised learning paradigms
Use Cases
Frequently Asked Questions
From the Blog
What Is Self-Attention in Neural Networks
Self-attention lets each token in a sequence attend to every other token in the same sequence, building context-aware representations that power transformer models.
Read More AI & TechnologyWhat Does Self-Employed Really Mean? A Practical Guide
Self-employed means you work for yourself rather than an employer, earning income directly from clients or your own business instead of a salary. This guide explains the types, tradeoffs, and skills that help.
Read More AI & TechnologySelf-Consistency Prompting: Sampling Answers and Voting
Self-consistency samples the same reasoning prompt several times at a non-zero temperature and takes the majority answer rather than trusting one chain. This covers when the technique helps, how to extract and compare answers reliably, the cost multiplier it imposes, and when a cheaper approach wins.
Read More AI & TechnologyWhat Is Computer Vision? Real-World Examples
Computer vision lets machines interpret images — from medical scans to self-driving cars, explained simply.
Read More