RAG
Retrieval-Augmented Generation (RAG) is a technique that combines a language model with an external retrieval system: relevant documents or passages are fetched from a knowledge base (often via vector search over embeddings) and inserted…
Definition
Retrieval-Augmented Generation (RAG) is a technique that combines a language model with an external retrieval system: relevant documents or passages are fetched from a knowledge base (often via vector search over embeddings) and inserted into the model's prompt as context before it generates a response. This grounds the model's output in up-to-date, verifiable, domain-specific information beyond its training data.
Overview
Language models have two structural limitations: their knowledge is frozen at training time, and they can only reason over information present in their training data or current context window. RAG addresses both by connecting the model to an external, updatable knowledge source at inference time. A typical RAG pipeline works in three steps. First, source documents (manuals, wikis, support tickets, codebases) are split into chunks and converted into vector embeddings, then stored in a vector database. Second, when a user asks a question, the query is also embedded and used to perform a similarity search (vector search) against the stored chunks, retrieving the most relevant passages. Third, those retrieved passages are inserted into the prompt alongside the user's question, and the language model generates an answer grounded in that retrieved context, often citing sources. RAG is popular because it is far cheaper and faster than fine-tuning a model to memorize new knowledge, and it keeps information current — updating the knowledge base immediately changes what the model can answer, with no retraining required. It also improves transparency, since answers can be traced back to specific source documents, and it reduces (though does not eliminate) hallucination by anchoring generation in retrieved facts. RAG systems introduce their own engineering challenges: chunking strategy (how documents are split affects retrieval quality), embedding model choice, retrieval ranking and reranking, handling queries that need information spanning multiple documents, and managing the tradeoff between context window size and retrieval precision. Advanced RAG variants add techniques like hybrid search (combining keyword and vector search), query rewriting, and multi-hop retrieval for complex questions.
Key Concepts
- Combines a language model with an external, searchable knowledge base at inference time
- Uses vector embeddings and similarity search to retrieve relevant context for a query
- Keeps answers current without retraining the underlying model
- Improves traceability by grounding answers in retrievable source documents
- Reduces, but does not eliminate, hallucination compared to a model relying only on parametric memory
- Requires infrastructure: a vector database, an embedding model, and a chunking strategy
- More cost-effective than fine-tuning for injecting large or frequently changing knowledge
- Can be combined with reranking and hybrid search for improved retrieval quality