Context Window

Definition

The context window is the maximum amount of text, measured in tokens, that a language model can process and 'see' at one time, encompassing both the input prompt and the generated output. Anything beyond this limit is truncated or dropped, meaning the model has no memory of information outside its current context window unless it is explicitly re-included.

Overview

Every interaction with a language model happens within a fixed-size window of tokens — the smallest units of text the model processes, roughly corresponding to word fragments. The context window determines how much conversation history, source documents, or instructions can be included in a single request. If a conversation or document exceeds this limit, earlier content must be dropped, summarized, or otherwise managed for the model to continue responding coherently. Context window size has grown dramatically as model architectures and hardware have improved, from a few thousand tokens in early models to context windows spanning hundreds of thousands or even millions of tokens in some modern models. A larger context window allows a model to process entire books, codebases, or long conversation histories without external retrieval, but it comes with tradeoffs: processing cost and latency generally scale with the amount of context used, and some research suggests model attention can degrade for information buried in the middle of a very long context (sometimes called the 'lost in the middle' effect). Because of context window limits, applications working with large amounts of information often use retrieval-augmented generation (RAG) to select only the most relevant chunks of a larger knowledge base to include in the prompt, rather than attempting to fit everything into context. Similarly, long conversations in chat applications are often summarized or truncated to stay within the window while preserving the most relevant history. Understanding context window limits is essential for building reliable LLM applications: prompt engineers and developers must budget tokens carefully, balancing how much context to include against cost, latency, and the risk of important information being overlooked.

Key Concepts

• Measured in tokens, not characters or words directly
• Includes both the input prompt and the model's generated response
• Content beyond the window limit is truncated and unavailable to the model
• Larger context windows enable processing of long documents or extended conversations
• Larger context generally increases inference cost and latency
• Model attention quality can vary across a very long context
• Often managed via summarization, truncation, or retrieval (RAG) in applications
• Varies significantly by model provider and version

Use Cases

Frequently Asked Questions

Related in AI Fundamentals