Grouped Query Attention
Grouped Query Attention (GQA) is a transformer attention variant in which multiple query heads share a single set of key and value heads, reducing the size of the KV cache and inference memory bandwidth compared to standard multi-head…
Definition
Grouped Query Attention (GQA) is a transformer attention variant in which multiple query heads share a single set of key and value heads, reducing the size of the KV cache and inference memory bandwidth compared to standard multi-head attention while retaining most of its quality.
Overview
Standard multi-head attention (MHA) gives every attention head its own independent query, key, and value projections. This maximizes representational flexibility but means the KV cache — the stored keys and values needed for autoregressive generation — grows proportionally to the number of heads, which becomes a major memory and bandwidth bottleneck at inference time, especially for large models serving long contexts and large batch sizes. Multi-Query Attention (MQA), proposed by Noam Shazeer in 2019, took the opposite extreme: all query heads share a single shared key/value head, drastically shrinking the KV cache but sometimes at a noticeable cost to model quality and training stability. Grouped Query Attention, introduced by Google Research in 2023 (the GQA paper by Ainslie et al.), is a middle ground: query heads are divided into groups, and each group shares one set of key/value heads. With, say, 32 query heads divided into 8 groups, GQA uses only 8 key/value heads instead of 32, cutting the KV cache by 4x, while retaining more per-group specialization than full MQA. The paper also showed that existing MHA-trained models can be converted to GQA cheaply, by mean-pooling existing key/value heads within each group and then fine-tuning briefly, avoiding a full retrain from scratch. GQA has become a standard architectural choice in modern open-weight LLMs because it directly attacks the KV-cache memory bottleneck described in the KV cache entry: smaller KV cache means more concurrent requests can be served per GPU, longer contexts fit in memory, and inference throughput improves, all with only a small, often negligible, quality trade-off compared to full multi-head attention. Models including LLaMA 2 70B, LLaMA 3, Mistral, and many others use GQA, typically tuned to a specific query-to-KV-head ratio (e.g. 4:1 or 8:1) chosen to balance quality against serving efficiency for that model's size and target hardware.
Key Concepts
- Multiple query heads share a smaller number of key/value heads
- Sits between full Multi-Head Attention and Multi-Query Attention on the quality/efficiency spectrum
- Directly reduces KV cache memory footprint proportional to the group size
- Existing MHA models can be 'uptrained' into GQA via head pooling plus light fine-tuning
- Improves inference throughput and maximum batch size on fixed GPU memory
- Standard in LLaMA 2/3, Mistral, and many other modern open-weight LLMs
- Query-to-KV-head ratio is a tunable architectural hyperparameter (e.g. 4:1, 8:1)
Use Cases
Frequently Asked Questions
From the Blog
How Transformers Work: Attention Explained Simply
Transformers process all words at once and use attention to weigh how much each word relates to every other, letting models capture context and long-range meaning.
Read More AI & TechnologyWhat Is Self-Attention in Neural Networks
Self-attention lets each token in a sequence attend to every other token in the same sequence, building context-aware representations that power transformer models.
Read More AI & TechnologyHow to Store and Query Metadata Alongside Embeddings
Design the payload before you index anything: flat, typed, low-cardinality fields for the things you will filter on, with tenant and permission keys applied server-side on every query. Then decide deliberately between pre-filtering and post-filtering, because that choice determines whether restrictive filters return empty results.
Read More AI & TechnologyQuery Rewriting and Expansion for Better Retrieval
Query rewriting turns what the user typed into what the index can actually match - resolving pronouns, expanding jargon, and splitting compound questions. This covers conversational rewriting, multi-query fan-out, hypothetical document embedding, and how to tell whether any of it is helping.
Read More