Sparse Attention
Sparse attention is a family of transformer attention mechanisms that restrict each token to attending only to a subset of other tokens, rather than the full sequence, reducing the quadratic compute and memory cost of standard…
Definition
Sparse attention is a family of transformer attention mechanisms that restrict each token to attending only to a subset of other tokens, rather than the full sequence, reducing the quadratic compute and memory cost of standard self-attention.
Overview
Standard self-attention in transformers computes pairwise interactions between every pair of tokens in a sequence, giving it O(n²) time and memory complexity with respect to sequence length n. This becomes a hard bottleneck as context windows grow into the tens or hundreds of thousands of tokens. Sparse attention techniques address this by having each token attend only to a carefully chosen subset of positions — for example, a local sliding window, a fixed stride pattern, a small set of global 'anchor' tokens, or a learned/routed subset — cutting the effective complexity toward O(n log n) or O(n) while preserving most of the modeling power. Early influential sparse attention designs include the Sparse Transformer (OpenAI, 2019), which combined strided and local attention patterns; Longformer, which mixed a sliding-window local attention with a handful of global tokens for tasks needing long-range awareness; and BigBird, which added random attention links on top of local and global patterns and proved the combination retains the theoretical expressiveness of full attention under certain conditions. More recent large language models use variants such as block-sparse attention or attention sinks (as in StreamingLLM), where a small number of initial tokens are always retained regardless of window position, stabilizing generation over very long or streaming sequences. Sparse attention is distinct from, but often combined with, hardware-level optimizations like FlashAttention (which speeds up exact attention via IO-aware kernels rather than skipping computation) and from grouped-query attention (which reduces KV-cache size rather than the attention pattern itself). In practice, sparse attention patterns must be chosen carefully: too aggressive sparsification can hurt a model's ability to do tasks requiring long-range dependency tracking, such as multi-hop reasoning or needle-in-a-haystack retrieval across long documents. Modern long-context LLMs like GPT-4, Claude, and Gemini use a mix of architectural tricks — sparse or windowed attention, efficient KV-cache management, and specialized positional encodings — to serve context windows of 100K to millions of tokens economically.
Key Concepts
- Reduces attention complexity from O(n²) toward O(n log n) or O(n)
- Common patterns include local/sliding-window, strided, global-token, and random links
- Enables practical long-context inference and training at lower compute cost
- Notable implementations: Sparse Transformer, Longformer, BigBird, StreamingLLM attention sinks
- Often combined with FlashAttention-style IO-aware kernels for further speedups
- Trade-off between compute savings and long-range dependency modeling accuracy
- Used alongside KV-cache compression techniques in production long-context LLMs
Use Cases
Frequently Asked Questions
From the Blog
How Transformers Work: Attention Explained Simply
Transformers process all words at once and use attention to weigh how much each word relates to every other, letting models capture context and long-range meaning.
Read More AI & TechnologyWhat Is Self-Attention in Neural Networks
Self-attention lets each token in a sequence attend to every other token in the same sequence, building context-aware representations that power transformer models.
Read More AI & TechnologyHow Large Language Models Actually Work
LLMs seem magical until you understand what they are: next-token predictors trained on massive text corpora. This guide explains tokenisation, embeddings, the transformer architecture, attention mechanism, and how training works — without requiring a maths degree.
Read More AI & TechnologyHow Transformers Work: The Architecture Behind Modern AI
Transformers are neural networks that use attention to weigh how words relate, letting AI process whole sequences in parallel and capture long-range context.
Read More