KV Cache
The KV cache (key-value cache) is a memory structure used during autoregressive transformer inference that stores the key and value projections of previously processed tokens so they can be reused rather than recomputed at each new…
Definition
The KV cache (key-value cache) is a memory structure used during autoregressive transformer inference that stores the key and value projections of previously processed tokens so they can be reused rather than recomputed at each new generation step.
Overview
When a transformer language model generates text autoregressively — one token at a time — computing attention for a new token requires comparing it against the keys and values of every prior token in the sequence. Without caching, generating each new token would require reprocessing the entire preceding sequence through every layer, an extremely wasteful O(n²) pattern of redundant computation over a full generation. The KV cache eliminates this redundancy: at each layer, the key and value vectors computed for a token are stored once and reused for every subsequent generation step, so each new token only requires computing its own query, key, and value and then attending against the cached keys/values of all prior tokens. This optimization is what makes practical, low-latency LLM text generation possible, but it comes at a real memory cost: the KV cache grows linearly with sequence length, batch size, number of layers, and number of attention heads, and for large models with long contexts it can consume more GPU memory than the model's weights themselves. This has driven significant research and engineering effort into KV-cache-efficient architectures and serving systems: Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) shrink the cache by sharing key/value heads across multiple query heads; quantized KV caches store keys/values in lower precision (e.g. INT8 or FP8); and serving systems like vLLM introduced PagedAttention, which manages the KV cache using OS-style virtual memory paging to eliminate fragmentation and enable efficient memory sharing across concurrent requests. KV cache management is now a central concern in LLM inference engineering — it directly determines the maximum batch size and context length a given amount of GPU memory can serve, and consequently drives the cost-per-token economics of running LLMs at scale. Techniques like prefix caching (reusing KV cache across requests sharing a common prompt prefix, e.g. a system prompt) are also widely used to cut latency and cost in production serving stacks.
Key Concepts
- Stores previously computed key/value projections to avoid recomputation during generation
- Turns autoregressive decoding from quadratic to roughly linear per-token cost
- Memory usage scales with sequence length, batch size, layers, and attention heads
- Grouped-Query and Multi-Query Attention reduce cache size by sharing KV heads
- Quantized KV caches (INT8/FP8) trade precision for reduced memory footprint
- PagedAttention (vLLM) manages KV cache memory using OS-style paging to reduce fragmentation
- Prefix caching reuses KV cache across requests sharing a common prompt prefix
Use Cases
Frequently Asked Questions
From the Blog
The KV cache explained: why it dominates LLM memory during serving
The KV cache makes generation fast and memory-bound at once. Learn what it stores, why it caps concurrency, and the levers that shrink it without hurting quality.
Read More Cloud & CybersecurityHow to cache dependencies in CI without breaking builds
Speed up pipelines safely: designing cache keys from lockfile hashes, choosing what to cache, restore-key fallbacks and spotting a poisoned cache.
Read More Cloud & CybersecurityDesigning a caching layer and avoiding cache stampedes
Add a cache without adding an outage: choosing what to cache, invalidation strategies, TTL jitter, single-flight rebuilds and handling cold starts safely.
Read More Data ScienceHow to build a tf.data pipeline that stops starving your GPU
Low GPU utilisation usually means the input pipeline cannot keep up. Confirm it by timing the pipeline alone, then order the operations correctly — map, cache, shuffle, batch, prefetch — and parallelise the expensive stages. Ordering affects correctness as well as throughput.
Read More