Two weeks ago Nix Nulty showed that an agent benchmark can be won by gaming the test. This week the number misbehaves without anyone cheating at all: OpenAI ran its own model against the same tasks twice and got 13.3% and 38.3%, depending only on how the harness was wired. The model never changed. If a single vendor’s own experiment can nearly triple a score by flipping two settings, the leaderboard gap you’re using to pick between two vendors deserves a much harder look. — Muximus
Here is a number OpenAI published about its own model, and it is the most useful thing a vendor has said about benchmarks all year. Run GPT-5.6 Sol against the public tasks in ARC-AGI-3 and it scores 13.3% on the benchmark’s official harness. Run it against the exact same tasks with two API settings switched on and it scores 38.3%. Same model. Same tasks. Nearly triple the score. The only thing that changed was the plumbing around the model.
Sit with that, because the entire ritual of quoting a benchmark number depends on nobody doing so. A leaderboard figure gets read as a property of the model - this one is 38% good, that one is 13%, pick the bigger number. OpenAI’s own data says the figure is a property of the model and the scaffolding that ran it: what memory it was allowed to keep, how its context got trimmed, how the task was fed to it. Rewire the scaffolding and the same model looks twice as sharp, or half as sharp, with nothing changed about the thing anyone claims to be measuring.
What actually moved
The two settings are not black magic. The first kept the model’s private reasoning between turns instead of wiping it after every move - so it stopped being handed the puzzle cold each time and asked to solve it again from scratch. The second swapped rolling truncation, which just deletes the oldest tokens as the window fills, for compaction, which condenses the history instead of dumping it. Both are choices a harness makes on your behalf, usually where you can’t see them.
And here’s the detail that kills the lazy explanation. The higher score used roughly six times fewer output tokens than the lower one. This is not “throw more compute at it and the number climbs.” It’s the same model generating less and scoring more, because the harness stopped throwing away the parts of its own thinking that were doing the work. The score went up because the wiring got smarter. The model didn’t.
Why this is the whole story, not a footnote
A leaderboard number is a bundle wearing a single price tag. Inside it sit the model, the harness, the memory policy, and the prompting - and only the model’s name makes it onto the slide in the board deck. Everything else stays back on the bench where the number was cooked.
Which is exactly why a small gap between two models on a public leaderboard is close to worthless for a buying decision. One model quoted at 40%, another at 35%: that five-point gap might be capability, or it might be that the first ran with retained reasoning and compaction and the second didn’t. OpenAI just demonstrated a 25-point gap opening on one model from harness settings alone. Next to that, your five-point vendor bake-off is measurement noise with a decimal point.
Credit where it’s due, and keep the numbers honest while giving it. These scores are graded against a human baseline - how efficiently the model acts compared to a person, not a raw count of puzzles cleared - and OpenAI pegs the average human tester at around 48%. Both machine figures, 13.3% and 38.3%, sit under that. So this is not the story where the model was secretly superhuman and a bad harness was hiding it. It’s the plainer, more useful story: the number you get handed is unstable, and the instability tilts toward whoever picked the harness.
And who picked the harness
The vendor did, which is the part to hold up to the light. Bolted to the finding is OpenAI’s recommendation: compare models under production-like settings, not a generic harness. As advice, it’s correct - a score from a configuration nobody would ever deploy tells you very little. It is also an interested party arguing that the setup which flatters its model is the fair one to grade on. Both of those are true at the same time, and a reader is owed both halves: the advice is sound, and the advisor gets paid when you take it.
None of this makes ARC-AGI-3 a bad benchmark or GPT-5.6 Sol a bad model - and OpenAI publishing its own embarrassing 13.3% instead of quietly shipping the 38.3% is more candor than the category usually offers. It makes the number a bad unit of comparison the moment it arrives stripped of the harness that produced it. That’s a sharper claim than “benchmarks are hype.” The benchmark measured something real. It measured the harness as much as the model, and the two got printed as one figure.
The Hype-o-Meter
The benchmark score as a standalone spec: 8/10 overhyped. Not because the tests are fake or the models are bad, but because the figure gets quoted as if it belonged to the model alone, and it demonstrably doesn’t. The reader-facing fix costs one question.
When a vendor puts a benchmark number in front of a decision, the number is half an answer. The other half is the harness. So the question that turns a slide back into evidence is: which configuration produced this - the memory policy, the context handling, the prompting - and does it hold up under the setup that would actually run in production? A vendor who can answer has handed over a measurement. A vendor who can’t has handed over marketing with a decimal point.
