What is in-context learning?

It's the ability of a large language model to learn a task pattern from examples or instructions placed directly in the prompt, adjusting its output without any change to its underlying weights.

How is in-context learning different from fine-tuning?

Fine-tuning updates a model's parameters through additional training on labeled data, producing a permanently adapted model. In-context learning changes only the model's behavior for a single inference call, based on the current prompt.

What are zero-shot, one-shot, and few-shot learning?

These describe how many task examples are given in the prompt: zero-shot provides only an instruction, one-shot provides a single example, and few-shot provides several examples before the actual query.

Why does in-context learning work?

The precise mechanism is still debated, but leading theories suggest large models implicitly infer which task is being requested based on patterns seen during pretraining, effectively performing a form of inference over latent tasks.

Does model size affect in-context learning ability?

Yes, larger models trained on more diverse data generally show much stronger and more reliable in-context learning, which is part of why the phenomenon became prominent with models like GPT-3.

What limits in-context learning?

It is constrained by the model's context window, can be inconsistent depending on example order and phrasing, and generally underperforms fine-tuning for narrow, high-stakes tasks requiring maximum reliability.

Is in-context learning the same as prompt engineering?

They're related but not identical. Prompt engineering is the practice of crafting prompts; in-context learning is the underlying model capability that prompt engineering techniques like few-shot examples exploit.

AI Fundamentals

In-Context Learning

IntermediateConcept6.7K learners

In-context learning is the ability of a large language model to adapt its behavior on a new task using only examples or instructions provided in the prompt, without any gradient updates to the model's weights.

#InContextLearning #AIFundamentals#Concept#Intermediate#ChainOfThoughtPrompting #PromptEngineering #RAG #FineTuning #ArtificialIntelligence#Glossary#SkillVeris

Definition

Overview

In-context learning (ICL) emerged as a striking property of large language models trained on next-token prediction over massive text corpora: given a handful of demonstrations of a task inside the prompt — such as a few input-output pairs for translation or classification — the model can infer the pattern and apply it to a new input, all within a single forward pass. This stands in contrast to the traditional machine learning workflow of fine-tuning, where a model's parameters are explicitly updated on labeled examples through backpropagation. ICL is typically described along a spectrum: zero-shot (only an instruction, no examples), one-shot (a single example), and few-shot (several examples). The phenomenon was popularized by the GPT-3 paper, which showed that scaling model and data size significantly improved few-shot performance without any weight updates. Researchers still debate the underlying mechanism — some frame it as implicit Bayesian inference over latent tasks seen during pretraining, others as a form of gradient descent simulated within the forward pass through attention layers. ICL is central to how most people interact with modern LLMs like GPT-4 or Claude: prompt engineering, few-shot prompting, and chain-of-thought prompting are all applications of in-context learning. It is also the foundation for retrieval-augmented generation, where retrieved documents are injected into the context so the model can reason over them without retraining. Because it requires no model updates, ICL makes LLMs dramatically cheaper and faster to adapt to new tasks, though it is bounded by the model's context window and can be less reliable than fine-tuning for highly specialized or high-stakes tasks.

Key Concepts

Adapts model behavior purely through prompt content, with no weight updates
Spans zero-shot, one-shot, and few-shot prompting styles
Scales in effectiveness with model size and pretraining data diversity
Bounded by the model's context window length
Underpins prompt engineering techniques like chain-of-thought and few-shot prompting
Enables rapid task adaptation without a training pipeline or labeled dataset
Performance is sensitive to example order, phrasing, and selection
Forms the mechanism that makes retrieval-augmented generation possible

Use Cases

Adapting an LLM to a new classification or extraction task with a handful of examples

Teaching a model a custom output format via demonstrations in the prompt

Rapid prototyping of NLP tasks without collecting a fine-tuning dataset

Injecting retrieved documents into context for grounded question answering

Steering tone, style, or persona through instructions and examples

Domain adaptation for low-resource tasks lacking sufficient training data

Frequently Asked Questions

From the Blog

AI & Technology

20 ChatGPT Prompts to Boost Your Productivity

Great prompts share four parts: role, task, context, and format — here are 20 ready-to-use prompts for daily work.

Prompt Engineering: Get Better Results from Any LLM

The difference between a mediocre AI output and an excellent one is usually the prompt. This guide covers the techniques that consistently produce better results: clarity, context, examples, chain-of-thought, system prompts, and output formatting — with real before/after examples.

Learn Python Through Cricket Statistics

Cricket generates rich data — runs, wickets, overs, strike rates, economy rates. This project uses real IPL-style match data to teach you pandas, matplotlib, and data analysis in a context that actually interests you. No dry tutorials — just cricket and code.

Learn SQL Through Football Data

Football generates rich match data — goals, assists, passes, xG, red cards. This project uses a Premier League dataset to teach SQL SELECT, WHERE, GROUP BY, JOIN, and HAVING in a context that makes every query meaningful rather than abstract.