ARC-AGI-3 agents stumble on raw grids: Sarbloh computes geometry on the host and lifts the hidden-set score from 0.27 to 16.53

Wasted actions on ARC-AGI-3 come mainly from misreading the scene rather than from weak planning: given the board serialized as integers, one frontier reasoning model spent 200 to 300 actions on the opening level of ls20 without clearing it. Rendering the same board as an image cut required exploration to 12 to 24 actions; adding an explicit breakdown of terrain, status gauges and color parity brought it to 12; letting the model express its search as executable Python cleared the level in 6 actions against a human baseline of about 10. The comparison held the prompt template and environment transitions fixed on the ls20 opening level and changed only the input representation in a single diagnostic trial with one model; over 90% of the efficiency gain came from restructuring perception rather than extending chain-of-thought deliberation.

The agent acts only from inside its Python kernel, submitting a path of up to 100 moves at once through await play(path, predict=f), with its own world model supplied as the prediction. Early builds split code execution and move submission into two tools, so the model modeled the game in one, sent a single move from the other and re-derived its state, costing many turns per plan; merging them moved the hidden-set score from 5.14 to 5.65. When a prediction disagrees with the next frame, play() logs the first mismatched coordinate and keeps going; the path stops only at level or game end, when consecutive identical actions collide without changing a pixel, or when the revisit guard sees a repeated board.

Each frame yields three views sharing one object-identity space: an object table with cross-frame identity tracking, a structured briefing that separates measured facts from guesses, and a spatial map annotated with affordances. An early text-only encoder that printed the lattice, palette and connected components scored 3 of 28 on the perception rubric while spending about 3,500 tokens per frame; adding role guesses with verification tests raised it to 20, and fusing code-computed relations raised it to 25. The fused briefing costs about 370 tokens per frame, 89.4% fewer than the baseline; across 23 held-out official games, screenshot-to-grid reconstruction reached 100% cell accuracy, while single-frame role guesses identified the agent in only 6 of 14 keyboard games.

The demonstrated setting is the official ARC-AGI-3 offline track: one NVIDIA RTX PRO 6000 GPU, no network access, a nine-hour wall-clock ceiling for the whole suite, 25 public games and an undisclosed hidden set. The intended reader is an engineer working on interactive agents and perceptual representations; the 143-of-183-levels run in 8.2 hours on the public set had the briefing and map switched off, so it characterizes the action interface, serving loop and scheduler rather than the perception engine.

The authors note that the perception engine evaluates role hypotheses independently at each frame without maintaining a unified Bayesian belief distribution across multi-step episodes; antecedent failure on complex perceptual transitions remains the main source of score loss; and the results rely on general-purpose open-weight models without task-specific policy fine-tuning. Run B with the briefing on differs from Run A with it off in more than the briefing and has one run per arm, so the authors report the drop in median action ratio from 0.77 to 0.68 as consistent with perception helping, not as a measured effect. The source is also inconsistent about the hidden-set gain: the text calls 16.53 a 13.7 efficiency improvement over the first Milestone winner, while the same table gives that winner a score of 1.21.