A new benchmark did something unusual: it ran eight coding agents against code they could not possibly have seen before, because it was licensed out of real companies and has never been on the internet. The headline number is 38.8%. Penny's interest is in the other column of the spreadsheet - the one recording how the failures failed. Almost none of them were bad code. They were agents building the wrong thing, in the wrong place, on a guess nobody checked. Which is a very different problem, with a very different fix. — Muximus
A benchmark published this month ran eight frontier coding agents against ten tasks taken from private company codebases. The best configuration passed 38.8% of its attempts, and one task was not completed once in 64 tries by any agent in the set.
The percentage is the part that will get quoted. The more useful column is the one next to it, where the benchmark recorded how each of the 468 failed runs failed. Sorted that way, the agents were rarely wrong about how to write code. They were wrong about what the code was supposed to do.
An electrician who has never seen this house
Imagine hiring an electrician with a spotless record. They can wire a socket in their sleep, and nothing they have ever installed has burned down.
Now send them into a house they have never been inside. The wiring is fine work, but it is this house's fine work: the kitchen ring shares a breaker with the garage freezer, the previous owner ran a spur to the shed that is not on any drawing, and there is a rule - unwritten, known to exactly two people - that the loft lights must stay on the upstairs circuit because of what happens to the boiler otherwise.
The electrician is not going to fail at wiring. They are going to fail at knowing this house.
That is the shape of what Real-SWE found, and it is worth about four paragraphs of your attention, because almost every number quoted about coding agents is measured in a house they have already lived in.
What was measured, and whose benchmark it is
Real-SWE comes from Specific Labs, a company that builds evaluations for this market and therefore has an interest in the answer. Specific Labs designed the tasks, ran them and graded them. The tasks are not published in full - the company offers access to a sample on request - and we found no independent reproduction of the run. That belongs beside every number below, including the ones that are about to sound damning.
What the company did is the genuinely new part. It licensed production codebases from real businesses and lifted actual engineering tasks out of them. The examples it describes include an events platform with more than 200,000 users and a consumer fintech service that processes more than 100,000 bank statements. Ten tasks, eight model-and-harness pairings, eight attempts at each task by each pairing: 640 scored runs.
Four words to get out of the way, because they are doing real work here and nobody should have to pretend they already knew them.
A rollout is one attempt. The agent gets the instruction, works in a sealed copy of the codebase, and stops.
A harness is the software wrapped around the model that hands it a terminal, file access and a loop to work in - Claude Code, Codex CLI, Gemini CLI. Real-SWE scores the pair rather than the model on its own, which is the honest choice, because nobody in a real company is using a model without one.
The resolution rate is the share of rollouts that passed, averaged over eight independent attempts per task. In benchmark papers this is written pass@1, which is a wonderful example of a term that exists to save four words and costs everyone a Google.
And the verifier is the test that decides pass or fail. It was drawn from the codebase's own test suite - in some cases those tests verbatim - and was injected only at grading time. So the grading was the company's real tests, not somebody's opinion of whether the work looked right.
Fable 5.1 running in Claude Code topped the table at 38.8%. GPT-5.6 Sol in Codex CLI came last at 16.2%.
The scores moved more between tasks than between models
Pool all eight configurations and score each task on its own and the ten split into two groups.
Four landed at or above 40%: "Multi-region sweep" at 67.2%, "API keys & environments" at 65.6%, "Entitlement overage lines" at 50.0% and "Customer identity migration" at 40.6%.
The other six came in below 15% - "Billing schedule migration" at 14.1%, "API token metering" at 12.5%, "S3 datastore measurement" at 10.9%, "Linearizable scan" at 4.7%, "Tax jurisdiction" at 3.1%, and "Analytics stream reducer" at 0 for 64.
Ten tasks is a small set, and every per-task figure rests on 64 attempts, so the error bars are wide; Real-SWE shows 95% confidence intervals on its headline table. The task names are also the authors' own labels, not a neutral taxonomy.
What survives those caveats is the shape of it. The gap inside a single agent is wider than the gap between agents. Fable 5.1, the top scorer, went 8 for 8 on two tasks and 0 for 8 on three others, in the same suite, under the same conditions. Grok 4.6 finished fifth overall and was the only configuration to solve "Customer identity migration" every single time, on a task where the leader managed three attempts out of eight.
There is a tidy story available in that split, and the list refuses to support it. It is not that the agents coped with generic infrastructure and came apart on company-specific business rules. "Tax jurisdiction" and "Billing schedule migration" are business rules, certainly. But "Linearizable scan" and "Analytics stream reducer" are hard distributed-systems work with no billing logic anywhere near them. The six low scorers have no subject matter in common at all.
What they have in common is a way of going wrong.
The failure log
Real-SWE sorted every failed run into five categories borrowed from an earlier benchmark's published taxonomy. In the authors' own words:
- A missed requirement "leaves out behavior the instruction requires."
- An integration error is the "right idea, wired into the surrounding system incorrectly."
- An unverified assumption "builds on a guess about the system instead of checking it in the workspace."
- A regression "breaks existing behavior while making the change."
- A wrong file delivers the work "somewhere the running application never calls, such as a one-off script."
Read that list again with the electrician in mind. Only one of those five is a complaint about the wiring.
Missed requirement was the largest single category for four of the eight configurations, and the largest across the whole grid: 190 of the 468 failed runs, by our count from the per-model figures Real-SWE publishes. For Grok 4.6 it accounted for 67.2% of its failures; for Kimi K3, 53.8%. Integration error was the leading failure for Gemini 3.8 Flash at 49.1%, and 136 failures overall. Unverified assumption led for GPT-5.6 Sol at 43.3%, and came to 118 overall.
Regression - breaking something that already worked - was the second-rarest failure in the entire grid. Eighteen runs out of 468. It stayed under 11% of failed runs for every single configuration, and it was exactly zero for three of them: GLM 5.3, Grok 4.6 and Kimi K3 never once broke working behavior while making a change. Wrong file was rarer still at six runs, and zero for six of the eight.
Which is to say: by the benchmark's own accounting, these agents mostly did not write code that blew up, and mostly did not knock over what was already standing. They built something competent that was not what had been asked for, or put it somewhere it would never run, or proceeded on a hunch about the codebase that nobody stopped to confirm.
One number, two questions
A coding score gets read as the answer to one question. It is really the answer to two, and they are not equally easy to measure - which is a shape this column keeps running into, most recently in a headline adoption figure where a number measured something other than the thing it was read as measuring.
The first is whether the model can write the change correctly.
The second is whether it can work out what the change has to do in the first place - which behavior is required, where in the system it belongs, and which of its assumptions about this particular codebase are simply wrong.
A benchmark built out of public repositories can test the first one properly. The second is much harder to test there, and Real-SWE's stated design premise is the reason: tasks on private codebases are, in the authors' phrase, "natively out of distribution," because neither the code nor its solutions have ever been on the public internet. Public repositories are, to a large extent, the material these models were trained on. Their conventions, their naming, the reason a particular service exists the way it does - on a public benchmark a model has plausibly absorbed all of it, and inside a company's own repository it has absorbed none of it.
Four of Real-SWE's five failure categories live on exactly that ground.
Whether that fully explains the gap is not something one benchmark with ten tasks can settle, and it should not be presented as settled. But it is the explanation the benchmark was built to test, and the failure log is consistent with it.
Two things that did not fix it
The obvious remedies are in the data, and neither one worked.
Letting the agent think longer did nothing at all. Of the rollouts that finished in under ten minutes, 71.4% failed. Give it ten minutes or more and the failure rate crept up rather than down, to 73.4%. The patient attempts were fractionally worse.
Spending more did not order the table either. Estimated cost per rollout ran from about $2.50 to about $6.96. The cheapest configuration in the set, Gemini 3.8 Flash at roughly $2.50, finished third at 31.2%. GLM 5.3 cost about twice as much, $5.12, and finished fourth at 28.8%. GPT-5.6 Sol was the second cheapest at $2.65 and finished last of all. The top scorer was also the most expensive, so cost is not irrelevant - it simply does not predict where anyone lands. Two of those estimates, for Grok 4.6 and Kimi K3, are flagged incomplete by the authors and may run higher.
The question to ask back
Next time someone slides a coding-agent score across the table - a vendor's benchmark, an internal pilot's pass rate, a figure on slide eleven - there is one question that separates what was measured from what is being decided:
When it failed, was the code wrong, or was the requirement missed?
The answer decides what to do next, and the two branches could not be further apart.
If the failures are regressions and work delivered into files the application never calls - broken code, in other words - that is a model-or-harness problem, and waiting for the next release is a rational thing to do. Somebody else is working on it.
If the failures are missed requirements, unverified assumptions and integration errors, that is a specification problem wearing a benchmark's clothes. Nobody is shipping a model that fixes it, because the missing information was never in the model's reach. In Real-SWE it was the second branch by a distance: 444 of the 468 failed runs sat in those three categories, and 24 in the other two.
If nobody in the room can answer the question at all, that is the finding. It means the pilot is recording pass rates and discarding the only data that would tell anyone what to do about them.
Either way, the score on its own is not yet about your codebase. It is the same trap as a time-savings claim that does not transfer because two people are reading different clocks: the measurement is honest, and it is answering a question nobody in this room asked. A pass rate collected on code a model has plausibly already read tells you something real about a capability. It tells you very little about what happens when the same agent walks into a system whose rules exist in your repository and in the heads of the four people who wrote it.
What this makes worth doing
Sorting failures into those five buckets takes an afternoon and needs nothing anyone does not already have: the categories are published, and any pilot that ran agents at all has the transcripts. It is worth the afternoon because it is the difference between a quarter spent waiting and a quarter spent working.
On the second branch, writing the requirement down properly stops being hygiene and becomes a measurable input - one of the very few variables in this entire picture that an organization sets for itself. The same is true of how discoverable a system's rules are from inside the repository, which is the thing "unverified assumption" is really measuring.
Every absolute number above has a shelf life of roughly a quarter. The models turn over that fast, and the next table will look better. How much of a company's own behavior is written down somewhere an agent can actually find it will not turn over at all. That part stays exactly as good, or as bad, as it was left.
Sources
- Real-SWE, Specific Labs: https://withspecific.com/benchmarks/real-swe
