No manual—and the score is not "how many puzzles solved"

ARC-AGI-3 is an ARC Prize benchmark: 135 hand-built mini-game environments, with 25 demos public so anyone can play at arcprize.org. One rule runs through everything: no manual. No object list, no rule text, no goal description, no intermediate reward scores.

Each step, the model gets only this: a 64×64 grid of cells colored 0–15, plus seven buttons it can press.

The only way through: act, guess the rules, revise when wrong. Press a key, watch what changes, infer what that key does, then test the next step.

That percentage is not "puzzles solved"

The ARC-AGI-3 score is RHAE (Relative Human Action Efficiency). It does not count how many tasks you clear. It compares steps: model steps to clear each level versus steps a first-time human needed on that same level—and the penalty is squared.

Three more scoring details matter if you want to read the numbers: each level's score is weighted by level index—later levels count more; early tutorial levels barely matter; a run's ceiling is capped by how far you got, so missing the last level blocks 100% no matter how efficient you were; the human baseline is a high-median first-timer across players on that level, so an ordinary person will not hit 100% either. OpenAI's estimated human average is 48%.

The model isn't dumb—its memory is wiped every step

If the score only punishes detours, where did Sol's detours come from? OpenAI reviewed several game logs. First impression matched ARC's: the model looked slow and unfocused, deliberating at length on each step and spinning in place. Digging further showed the model was fine—the harness was not. Two issues.

First: after every step, the private reasoning the model wrote for itself was discarded wholesale. On the next step, Sol had to re-learn the game from scratch. It could see which keys it had pressed and short notes, but not why it pressed them, what it had already figured out, or what it planned next—all gone.

Second: the harness used rolling truncation. Once context crossed a limit, the oldest messages were dropped. So past ideas could not stick around—and even the record of past keypresses was slowly pushed out of the window.

Both issues show up in ARC's open-source harness

ARC open-sourced the evaluation harness on GitHub (ARC-AGI-3-Agents; latest commit May 28, 2026). We pulled it and checked. Both problems are in the code.

First: every OpenAI call in the repo goes through chat.completions.create—20 call sites total. Zero responses.create, zero previous_response_id. That is stronger than "a setting was left off": the older API cannot structurally carry encrypted reasoning into the next turn. Dropped reasoning is not a missed flag—pick that API and the outcome is baked in.

Second: the message stack is true FIFO—oldest first to go:

def push_message(self, message):
    """Push a message onto stack, store up to MESSAGE_LIMIT with FIFO."""
    self.messages.append(message)
    if len(self.messages) > self.MESSAGE_LIMIT:
        self.messages = self.messages[-self.MESSAGE_LIMIT :]

When context fills, compress into a summary—don't delete the oldest

If the root cause is forgetting, the fix is two moves: let it remember, and stop freeing space by deleting.

OpenAI's models are trained to write private reasoning first, then answer or call a tool—and those reasoning messages are meant to stay in the dialogue history. When history no longer fits, compress and keep going. That is how ChatGPT and Codex are deployed. To match that production setup, they rewrote the ARC-AGI-3 harness on the Responses API.

Switch 1: retained reasoning—pass back the previous response ID

This switch is retained reasoning. On the Responses API, feed the previous response's ID (previous_response_id) into the next request and reasoning is kept automatically across tool calls and turns. Two things change: each step takes less thinking time, because the model does not need to re-learn the game; and it actually learns—strategies stay coherent instead of restarting every step.

Switch 2: replace rolling truncation with compaction

compaction is context compression. Per OpenAI's API docs, set context_management's compact_threshold on the request: once rendered tokens cross that threshold, the server runs a compression pass, packing prior state and reasoning into an encrypted compact item, then continues. That item is a black box to humans, but it carries what the conversation needs next.

The difference in one line: rolling truncation deletes the oldest records when memory is full—gone for good; compaction writes those records into a condensed form, frees space, and keeps writing. One truly forgets. The other keeps an abbreviated memory.

OpenAI's own diagram shows this mechanism across a real multi-step run:

Rolling truncation has a secondary side effect: the model spends most of its time in a nearly full context window, and fuller windows slightly hurt performance. Compaction empties the window again and sidesteps that too.

All five reasoning levels move up—~12× fewer tokens at the same score

The chart below plots both harnesses: x-axis is output tokens per game, y-axis is score. Each line has five points for low, medium, high, xhigh, and max reasoning.

The shapes tell the story at a glance: the official line crawls right—2.9M tokens per game to hit 13.3%; the new line climbs almost up the y-axis—0.49M tokens per game at max for 38.3%.

All five levels move up—first takeaway. OpenAI's own summary: at max, about 3× the score on about one-sixth the output tokens.

Go back to the scoring rule and the two ledgers separate cleanly: RHAE does not count thinking steps, so 13.3% → 38.3% is pure "fewer wrong turns"—remember your own judgments and you stop replaying ruled-out routes. Lower output tokens means you no longer re-learn the game from scratch every step. One change, two non-overlapping gains.

OpenAI also shipped a 4:29 animation that follows Sol's 175K context window across a sequence of levels:

ARC's own analysis points somewhere else

OpenAI says this deeper look was sparked by an ARC analysis. Published May 1, that piece reviewed 160 game replays and reasoning traces and landed elsewhere: the model never builds an understanding of the game. Same material; two readings side by side.

ARC's closing contrast: Opus 4.7 compresses into a confident wrong theory; GPT-5.5 never compresses at all.

They didn't fail because they didn't observe things. They failed because they couldn't ground those observations into a world model.

ARC Prize, "Analyzing GPT-5.5 and Opus 4.7 with ARC-AGI-3," May 1, 2026

The two stories are not mutually exclusive—forgetting last step's judgment makes a coherent world model harder. They do point at different places: one at the harness, one at the model.

Two more facts belong together. First, ARC's own footnote in that piece: under default testing GPT-5.5 does not return reasoning traces, so the qualitative analysis used a special setup—the official score does not come from that mode. That confirms from ARC's side that missing reasoning is a real phenomenon.

Second, ARC's July results page for the GPT-5.6 line is quite positive on Sol itself: the only competitive model of its generation, able to read unfamiliar scenes in the game's own vocabulary, replan when hypotheses fail instead of thrashing, and the first model to truly win a full public ARC-AGI-3 game (id ft09, 87%). "Looked not very smart" was OpenAI's impression from replays—not ARC's verdict on Sol.

Push the harness rewrite further and the numbers can climb much higher.

New API, keep the model's thinking, compact instead of delete

For practitioners, OpenAI's advice to API developers is three lines—no expansion in the original.

The concrete fields look like this.

previous_response_id
context_management.compact_threshold
store=false
POST /responses/compact

Their closing claim: benchmarks rarely measure the model alone. They also measure a pile of less-visible choices—API settings, harness design, prompts. And this is not the first time: they have seen public scores look oddly low before, only to find the eval's generic harness was dropping reasoning messages.

We hope these experiments are a reminder: benchmarks rarely measure the model alone. They also measure a whole set of less-visible choices—API settings, harness design, and prompts.

Ilan Bigio, Ted Sanders, OpenAI, July 29, 2026

One step further: Sol's score on this benchmark runs from ARC-verified 13.33%, to OpenAI's self-run 38.3%, to Schema's self-reported 95.35%—with the model weights never moved. Human baseline: 48%. Lined up like that, "how capable is this model?" still does not have a single settled answer.

ARC-AGI-3 is ARC Prize's puzzle-game eval. No manual at all: no rules text, no goal description, no intermediate reward. Each step the model gets only a 64×64 color grid plus seven pressable keys. Clear a level by pressing, guessing the rules, and revising when wrong.

The score is RHAE (Relative Human Action Efficiency). It compares steps: the model's steps to clear each level versus a first-time human on the same level, with a squared penalty on the extra steps. That percentage has nothing to do with "how many questions got right."

One more rule shapes every number below: actions that don't change the game screen don't count as steps — thinking, tool calls, retries, none of them. A rising score means fewer wasted moves. Saved output volume is a separate ledger.

OpenAI dug through a few match logs. First impression: the model looks dull — long pauses each step, looping in place. Dig further and the fault sits in the runner (the industry word is harness: the external code that feeds frames to the model, sends back chosen actions, and decides how much history it sees).

After every step, the private reasoning the model wrote for itself was thrown away wholesale. Once context filled up, even the keypress history was trimmed from the oldest end. The next step had to re-learn the game from scratch. What stayed in its hands:

The bug is forgetting. The fix is two moves: keep what it thought, and stop freeing space by deleting.

All five reasoning-effort tiers moved up. OpenAI's own summary at max: roughly 3× the score and about one-sixth the output tokens (model output, billed by volume). The more useful cell is same-score: