KVCache
The KV cache (key-value cache) is a memory structure used during autoregressive transformer inference that stores the key and value projections of previously processed tokens so they can be reused rather than recomputed at each new generation step.
6 resources across 1 library
Glossary Terms(6)
Vector Embedding Model
A vector embedding model is a neural network trained to map discrete inputs — words, sentences, images, or other objects — into dense numerical vectors such th…
Sparse Attention
Sparse attention is a family of transformer attention mechanisms that restrict each token to attending only to a subset of other tokens, rather than the full s…
Flash Attention
FlashAttention is an IO-aware exact attention algorithm that reorders and fuses the computation of transformer self-attention to minimize reads and writes to G…
Rotary Positional Embedding
Rotary Positional Embedding (RoPE) is a technique for injecting token position information into transformer attention by rotating query and key vectors in mult…
KV Cache
The KV cache (key-value cache) is a memory structure used during autoregressive transformer inference that stores the key and value projections of previously p…
Grouped Query Attention
Grouped Query Attention (GQA) is a transformer attention variant in which multiple query heads share a single set of key and value heads, reducing the size of…