ARC benchmark

Definition

The ARC benchmark, short for Abstraction and Reasoning Corpus, is a dataset of visual puzzle tasks designed to test general fluid intelligence and abstract pattern reasoning rather than memorized knowledge. Each task presents a small number of input-output grid pairs that demonstrate a hidden transformation rule, and the solver must infer that rule and apply it correctly to a new, unseen input grid, without any natural-language description of the rule provided.

Overview

ARC was designed specifically to resist the kind of pattern-matching that lets large models do well on knowledge-heavy tests without demonstrating genuine reasoning. Rather than asking a question with a known answer somewhere in a training corpus, each task presents a small handful of colored-grid examples illustrating a transformation that a human can usually infer within seconds, but that a system has to derive from scratch since every task uses a novel rule. Mechanically, a task gives two or three example pairs of input and output grids, each grid a matrix of a limited palette of colors, and the solver must produce the exact output grid for one or more held-out test inputs by inferring the shared transformation, such as reflecting a shape, completing a pattern, or recoloring based on a rule. Scoring is exact match on the output grid, so partial understanding does not earn partial credit, which makes ARC an unusually strict test of correct generalization. Among reasoning benchmarks, ARC differs from knowledge tests like MMLU and word-problem sets like GSM8K by removing language and factual recall almost entirely, isolating abstract visual reasoning instead. It is closer in spirit to human IQ-style pattern tests than to typical NLP benchmarks, and it was built explicitly to remain hard for large language models even as they mastered many text-based reasoning tasks. In practice, ARC is used by AI labs and researchers to probe whether a system can generalize to genuinely novel problems rather than interpolate within its training distribution, and it underlies a public prize competition that has driven a wave of specialized program-synthesis and neuro-symbolic approaches aimed at solving it. It is also used as a stress test for claims about emergent general reasoning in large models. Its main limitation is that strong performance often comes from systems built specifically around ARC's grid format and transformation types, so success does not necessarily transfer to reasoning in other domains. The benchmark is also narrow in scope, testing visual-spatial pattern inference rather than the full breadth of what would be considered general reasoning, and its small official test set makes statistical comparisons between close scores less reliable. A further complication is that human performance on ARC is itself uneven across task types, so comparisons between human and machine scores need care, and the prize competition's evolving public and private test splits mean year-over-year leaderboard numbers are not always directly comparable across different competition cycles. Researchers citing ARC results should therefore specify which version of the dataset and which split they used, since successive editions of the benchmark have adjusted task selection to keep pace with solvers that had started to overfit to earlier releases.

Key Features

• Grid-based puzzle tasks with a hidden transformation rule per task
• Each task uses a novel rule, resisting memorization from training data
• Scored by exact match on the predicted output grid
• Minimizes reliance on language or factual world knowledge
• Anchors a public prize competition driving new solving approaches
• Designed to remain hard even for large, knowledge-strong models
• Closer to abstract pattern-IQ tests than typical NLP benchmarks

Use Cases

Alternatives

Frequently Asked Questions

Related in AI Models