ARC-AGI-2 is the second generation of the Abstraction and Reasoning Corpus, a benchmark built to test whether a system can work out a rule it has never seen before and apply it correctly. A task is a set of small colored grids. You get a handful of input and output pairs that demonstrate some transformation, then a new input grid, and you have to produce the output grid the same rule would give. The rules are things like “move every shape until it touches the wall” or “fill the enclosed region with the color of its border,” but they are never stated, and every task uses a different one.
That design is the point. Most benchmarks reward a model for having absorbed a fact or a pattern during training. ARC-style tasks are constructed so that memorization does not help, because the specific rule in front of you does not appear anywhere in a training corpus. What gets measured is closer to fluid reasoning: how efficiently a system can acquire a new skill from a few examples. The original ARC was introduced by François Chollet as part of an argument that intelligence should be measured as skill acquisition efficiency rather than as accumulated skill, and ARC-AGI-2 continues that line.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- ARC-AGI-2 presents grid transformation puzzles where the rule must be inferred from a few demonstration pairs and applied to a new input.
- Scoring is exact match on the produced grid. There is no partial credit and no fuzzy grader, which makes it one of the few benchmarks with no scoring ambiguity.
- Tasks are calibrated against human testing, with each one confirmed solvable by more than one person under test conditions. That is a floor, not a claim that people find them easy.
- Direct-answer prompting has historically scored very poorly, which is the benchmark working as intended. The picture is moving, so check the current leaderboard rather than trusting any snapshot, including this one.
- A score on ARC-AGI-2 says something about general reasoning under controlled conditions. It says nothing about whether your agent handles your tools, your data, and your users.
How an ARC-AGI-2 task is structured
Every task follows the same format. A small number of demonstration pairs establish a transformation, and one or more test inputs ask you to apply it. Grids are small and use a limited palette of colors, so the search space of possible outputs is finite but the space of possible rules is not.
The rules draw on what the benchmark’s authors describe as core knowledge priors: basic notions that humans acquire early and apply without instruction. Objectness, meaning that contiguous blocks of color behave as things. Counting. Symmetry and rotation. Simple notions of containment, gravity, and contact. A task might combine two or three of these in a way that is obvious once you see it and invisible until you do.
Because the answer is a grid, grading is mechanical. Your output either matches the reference cell for cell or it does not. Submissions are generally allowed a small number of attempts per task, and a task counts as solved if an attempt matches exactly. That removes the grader reliability problem that shows up in most modern evaluation work, where a model or a heuristic has to judge free-form text and can be wrong. If you have ever had to calibrate a judge against human labels, the appeal of an exact-match target is obvious.
What the second generation changed
ARC-AGI-2 was built after the original benchmark had been studied for years, and the revisions respond to how systems were actually beating individual tasks. The broad goals were to keep tasks within reach for people while making them resistant to the strategies that had started to work: brute-force enumeration of candidate programs, pattern matching against previously seen ARC tasks, and simply spending enormous amounts of compute per task. Difficulty is calibrated against human testing, with each task confirmed solvable by more than one person under test conditions. Cost per task is treated as part of the result rather than as an implementation detail, because a system that gets an answer by burning unbounded compute has not demonstrated efficient reasoning.
The practical effect is that a system can no longer look good by being large. It has to form a hypothesis about the rule, apply it, and check itself. Direct-answer prompting has historically scored very poorly, which is the intended behavior of the benchmark rather than a flaw in it. The picture is moving: reasoning models that spend more inference compute on a single problem have narrowed the gap, and program-synthesis approaches that generate and test candidates remain competitive. Check the current leaderboard rather than trusting any snapshot, including this one.
Why language models find this hard
A model trained to predict text is extremely good at retrieving and recombining patterns it has seen. ARC-AGI-2 tasks are engineered so that there is nothing to retrieve. The model has to build a small theory from three or four examples and commit to it, then render the result precisely, cell by cell, with no tolerance for an off-by-one placement. Direct-answer prompting rarely produces that on its own. What helps is having some way to test a candidate rule against the demonstration pairs before committing to it.
That is the same architectural idea as an agent loop, where the model proposes, something external verifies, and the loop repeats. The tradeoffs are discussed in the handbook on how AI agents are built.
What a benchmark score does not tell you
ARC-AGI-2 is a clean instrument for one narrow question: can this system reason about novel abstract structure. It was never meant to predict whether an application built on that system will hold up. Public benchmarks and leaderboards compress a lot of context into one number. The gap widens once agents enter the picture, because the environment becomes part of the measurement.
Your own system needs its own measurement: traces of real sessions, criteria written for your tasks, and a way to catch regressions when a prompt or model version changes. That practice is laid out in agent evaluation.
FAQ
How is ARC-AGI-2 scored?
By exact grid match, with no partial credit and no judge model involved. A small number of attempts per task is typically permitted, and the reported figure is the fraction of held-out tasks solved.
Why do language models score so poorly on it?
Because the tasks are designed to defeat retrieval. Each rule is novel, the demonstration set is tiny, and the output must be exact. A model producing an answer in a single pass has no mechanism to test its hypothesis before committing, which is the gap that search, program generation, and extra inference compute are all trying to close.
What is the difference between ARC and ARC-AGI-2?
Same format and same underlying idea, tightened. The second generation targets tasks that stay within reach for humans while resisting brute-force search and memorization of earlier ARC tasks, and it treats compute cost per task as part of the evaluation rather than as something to ignore.
Does ARC-AGI-2 measure AGI?
No. It measures one property, generalization to novel abstract tasks, that most people consider necessary for general intelligence but not sufficient. Saturating it would be evidence of real progress on that property. It would not settle anything about planning, long-horizon reliability, or acting in the world.
Is it useful for choosing a model for my application?
Only as a weak signal about novel reasoning under controlled conditions. For anything you plan to ship, the useful evidence comes from your own evaluation set and from watching real behavior, which is what an agent observability platform is for.
