Grouped Query Attention
Grouped Query Attention (GQA) is a transformer attention variant in which multiple query heads share a single set of key and value heads, reducing the size of the KV cache and inference memory bandwidth compared to standard multi-head attention while retaining most of its quality.
5 resources across 1 library
Glossary Terms(5)
Sparse Attention
Sparse attention is a family of transformer attention mechanisms that restrict each token to attending only to a subset of other tokens, rather than the full s…
Flash Attention
FlashAttention is an IO-aware exact attention algorithm that reorders and fuses the computation of transformer self-attention to minimize reads and writes to G…
Rotary Positional Embedding
Rotary Positional Embedding (RoPE) is a technique for injecting token position information into transformer attention by rotating query and key vectors in mult…
KV Cache
The KV cache (key-value cache) is a memory structure used during autoregressive transformer inference that stores the key and value projections of previously p…
Grouped Query Attention
Grouped Query Attention (GQA) is a transformer attention variant in which multiple query heads share a single set of key and value heads, reducing the size of…