GSM8K
By OpenAI
GSM8K is a benchmark dataset of grade-school-level math word problems used to evaluate the multi-step reasoning ability of language models. Each problem requires a short chain of arithmetic operations, such as addition, subtraction,…
Definition
GSM8K is a benchmark dataset of grade-school-level math word problems used to evaluate the multi-step reasoning ability of language models. Each problem requires a short chain of arithmetic operations, such as addition, subtraction, multiplication, or division, to reach a final numeric answer, and problems are written in natural language rather than symbolic notation. Models are typically scored on whether their final answer exactly matches the reference solution.
Overview
GSM8K was created to test something simpler-sounding benchmarks missed: whether a model can carry out a short sequence of dependent reasoning steps correctly, not just recall a fact or pattern-match a single calculation. Grade-school arithmetic is easy for a human with a pencil, but a model has to parse the narrative, decide what to compute at each step, and keep intermediate quantities straight, which exposes reasoning failures that a single-step math question would hide. Mechanically, the dataset consists of thousands of word problems, each solvable in two to eight steps using basic arithmetic, with a reference chain of calculations and a final numeric answer. Evaluation usually involves prompting the model to show its work, often through chain-of-thought prompting, and then checking whether the final number matches. Because the correct answer requires getting every intermediate step right, GSM8K is sensitive to compounding errors in a way that single-fact quizzes are not. Among reasoning benchmarks, GSM8K sits at an easier tier than harder math suites like MATH or competition-level problem sets, and it is more narrowly arithmetic than broad-knowledge tests like MMLU or abstract-pattern tests like the ARC benchmark. It became a common early proxy for whether prompting techniques such as chain-of-thought actually help, since the gap between direct-answer and step-by-step prompting on GSM8K was historically large. In practice, GSM8K is used to report reasoning capability in model cards, to compare prompting strategies, and to check whether fine-tuning or distillation has preserved a model's ability to chain steps together rather than just its surface fluency. It is also used as an ablation tool in research on reasoning methods, since its short, checkable problems make failure modes easy to inspect. Its main limitation is that current frontier models now score very highly on it, so it no longer discriminates well between strong models, and its problems are simple enough that memorization of similar templates during training can inflate scores without reflecting general reasoning ability. It also only covers grade-school arithmetic, so it says nothing about a model's ability to handle proofs, algebra-heavy problems, or reasoning outside of simple word problems. Because of this ceiling effect, many evaluators now treat a near-perfect GSM8K score as a baseline expectation rather than a differentiator, and instead look to harder, more varied math and logic suites when they need to distinguish between models that all clear GSM8K comfortably. Some later analyses have also flagged small ambiguities or errors in a handful of reference solutions, a reminder that even a widely used, carefully constructed benchmark benefits from ongoing scrutiny rather than being treated as a permanently fixed ground truth.
Key Features
- Thousands of grade-school word problems needing multi-step arithmetic
- Reference solutions include the intended calculation chain, not just an answer
- Commonly paired with chain-of-thought prompting during evaluation
- Exposes compounding reasoning errors across dependent steps
- Used widely to compare prompting strategies on the same model
- Easier tier than benchmarks like MATH or competition mathematics
- Scores have become saturated among top frontier models