Why can't the model just recompute everything each token without a cache?

It could, but it would redo identical work at every step — for a 1,000-token generation, that means the model reprocesses the growing prefix a thousand times, which is enormously wasteful compared to caching and reusing the unchanged keys/values.

Why does the KV cache use so much memory?

Its size scales with sequence length × batch size × number of layers × number of heads × head dimension × 2 (for key and value), which for large models with long contexts and many concurrent requests can exceed the memory used by the model weights themselves.

How does Grouped Query Attention reduce KV cache size?

GQA has multiple query heads share a smaller number of key/value heads, so fewer distinct key/value vectors need to be stored and computed, directly shrinking the cache proportionally to the reduction in KV heads.

What is PagedAttention?

A memory management technique from the vLLM project that allocates KV cache in fixed-size non-contiguous blocks, similar to virtual memory paging in operating systems, which reduces memory fragmentation and lets multiple requests efficiently share cached prefixes.

What is prefix caching?

It's the practice of reusing the KV cache computed for a shared prompt prefix (like a system prompt or few-shot examples) across multiple requests, avoiding redundant computation and cutting time-to-first-token for subsequent requests.

Does quantizing the KV cache hurt output quality?

Some precision loss is possible, but well-implemented INT8 or FP8 KV cache quantization is generally designed to have minimal impact on generation quality while substantially cutting memory usage, making it a common production trade-off.

AI Fundamentals

KV Cache

IntermediateTechnique6.4K learners

The KV cache (key-value cache) is a memory structure used during autoregressive transformer inference that stores the key and value projections of previously processed tokens so they can be reused rather than recomputed at each new…

#KVCache #AIFundamentals#Technique#Intermediate#GroupedQueryAttention #FlashAttention #SparseAttention #RotaryPositionalEmbedding #ArtificialIntelligence#Glossary#SkillVeris

Definition

Overview

When a transformer language model generates text autoregressively — one token at a time — computing attention for a new token requires comparing it against the keys and values of every prior token in the sequence. Without caching, generating each new token would require reprocessing the entire preceding sequence through every layer, an extremely wasteful O(n²) pattern of redundant computation over a full generation. The KV cache eliminates this redundancy: at each layer, the key and value vectors computed for a token are stored once and reused for every subsequent generation step, so each new token only requires computing its own query, key, and value and then attending against the cached keys/values of all prior tokens. This optimization is what makes practical, low-latency LLM text generation possible, but it comes at a real memory cost: the KV cache grows linearly with sequence length, batch size, number of layers, and number of attention heads, and for large models with long contexts it can consume more GPU memory than the model's weights themselves. This has driven significant research and engineering effort into KV-cache-efficient architectures and serving systems: Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) shrink the cache by sharing key/value heads across multiple query heads; quantized KV caches store keys/values in lower precision (e.g. INT8 or FP8); and serving systems like vLLM introduced PagedAttention, which manages the KV cache using OS-style virtual memory paging to eliminate fragmentation and enable efficient memory sharing across concurrent requests. KV cache management is now a central concern in LLM inference engineering — it directly determines the maximum batch size and context length a given amount of GPU memory can serve, and consequently drives the cost-per-token economics of running LLMs at scale. Techniques like prefix caching (reusing KV cache across requests sharing a common prompt prefix, e.g. a system prompt) are also widely used to cut latency and cost in production serving stacks.

Key Concepts

Stores previously computed key/value projections to avoid recomputation during generation
Turns autoregressive decoding from quadratic to roughly linear per-token cost
Memory usage scales with sequence length, batch size, layers, and attention heads
Grouped-Query and Multi-Query Attention reduce cache size by sharing KV heads
Quantized KV caches (INT8/FP8) trade precision for reduced memory footprint
PagedAttention (vLLM) manages KV cache memory using OS-style paging to reduce fragmentation
Prefix caching reuses KV cache across requests sharing a common prompt prefix

Use Cases

Enabling fast, low-latency token-by-token text generation in LLM serving

Determining maximum concurrent batch size for a given GPU memory budget

Prefix/system-prompt caching to reduce latency and cost in chat applications

Long-context serving where cache size becomes the primary memory bottleneck

Speculative decoding and other inference optimizations that build on cached state

KV Cache

Definition

Overview

Key Concepts

Use Cases

Frequently Asked Questions

Related Reading

KV Cache

Definition

Overview

Key Concepts

Use Cases

Frequently Asked Questions

Related in AI Fundamentals

Related Reading