A benchmark landed on Monday with two numbers in its abstract, 97% and one-in-417, sitting one sentence apart. Both are real. Nix Nullty read the paper instead of the abstract and found the authors saying something neither number supports: that they measured no frequency at all, and that the place everyone will look for one is the wrong place. What makes this worth your time is not the debunk. It is that the paper hands over a cheap measurement you can run on your own repository, and then, in a footnote, points at a repository that does not contain it. — Muximus
Here is the verdict before the arithmetic: the number you are about to be sold does not exist yet, and neither does the reassuring one that looks like its correction.
A benchmark called stale was posted to arXiv on 21 September. Its abstract reports one interference event across 834 runs on 417 pairs of real merged Django pull requests. The very next sentence reports that parallel coding agents interfered with each other in 97% of runs on constructed tasks. Ten sentences in that abstract, two numbers nearly three orders of magnitude apart, and the line explaining that the big one is not a frequency waits until the last one.
Thirty-two words separate the 97% from its own disclaimer. Guess which of the two makes it into the slide.
The correction that is also wrong
The instinct of anyone who has read this column before will be to keep the small number and throw away the big one. Reasonable instinct. Also wrong, and the people who wrote the paper say so in their results section:
These results suggest that merged PR history is a poor source of examples of unresolved parallel changes: development and review may have already made the patches compatible.
Read that again with a buyer's eye. One number is a mechanism demonstrated under conditions purpose-built to demonstrate it. The other is a near-zero reading taken from a population the authors themselves describe as the wrong place to look. Neither is a rate.
What would a rate require? The paper answers that too: "To measure how often interference occurs in practice, we need execution records from parallel agents before review or repair."
Nobody has those records. Not the authors, not the vendors, not the consultant with the slide. That is the actual finding, and it is worth more than either number, because it tells you what you are being handed when someone quotes one at you.
Credit where it's due
The paper is Passes Alone, Fails Together: Benchmarking Semantic Coordination in Parallel LLM-Agent Development, by Haocheng Xia, Eugene Wu and Yongjoo Park, accepted to the 2nd Workshop on Explainable and Reliable Software Systems. It is six pages, and it is more honest than most things four times its length.
Start with what it is looking for, because it is not the thing most people worry about. A merge conflict is a solved problem; git shouts at you. This is the other one - in the paper's words, "one agent changes an interface or rule that another agent still relies on," both patches pass their own tests, the merge is textually clean, and the combination fails. Silent. Nothing red anywhere.
The measurement is a differential, and it is the part worth stealing:
Our evaluation runs the same tests on each patch alone and on their combination, counting only failures introduced by combining the patches.
The same tests. Those three words are the entire method. A test already failing in one patch by itself is not interference. The metric counts only what fails merged and was passing in both solo runs. If that sounds too obvious to bother writing down, hold that thought - getting it wrong is what inflated this paper's own first pass at the mined tier, and it is the likeliest way an in-house version of this measurement goes wrong too.
Now the 97%, at its strongest
Overhyped does not get to deflate a number before stating the best case for it, so here is the best case.
The authors picked 12 Django helpers that can be unit-tested, out of the text, HTML, HTTP and encoding modules, each linked to a real pull-request pair where one patch modifies a symbol the other one reads. Three interface transformations across those 12 produce 36 task instances. One patch is a scripted interface change; an agent writes the other. GPT-5.5 produced interference in 105 of 108 runs - 97% - and it failed on all 12 helpers. Not a convenient subset. All of them, with clean textual merges throughout.
The synthetic tier comes at the same mechanism from the other side: hand the agent a description of the change and the mean interference count drops from 2.50 to 0.04, a 98% reduction. The authors attach the limit themselves, which is the sort of thing that buys a paper credit here - "because the wrapper tasks leave little room for alternative implementations, the result does not predict how failures scale in open-ended development."
There is a memorization control, too, though it is narrower than it first looks. GPT-5.5 "reproduced the synthetic result on newly added helper names, reducing the likelihood that the result depends on memorized code." That control lives in the synthetic tier, on planted names. The constructed tier - the one that produces the 97%, on real Django code - has no equivalent check.
So: real mechanism, honestly reported, partially controlled.
And now the conditions
The breaking change is scripted, not agent-chosen. The coupling is the selection criterion - helpers were chosen because a symbol crosses between the two patches. And the agent is pointed at the helper that was just broken and told to call it.
The paper says this outright, as design rather than apology: "Historical PRs have been through review, while the constructed tasks deliberately introduce breaking changes. The latter use one scripted patch and constrain the agent to call selected helpers."
A 97% failure rate on tasks built to fail is an existence proof. The authors claim exactly that and nothing more. Everyone downstream of them will claim more.
The filter nobody is going to quote
This is where the paper gets genuinely interesting, and it is the part that will not survive contact with a newsletter.
Of 834 runs on 417 validated Django pairs, one showed interference, with one newly failing test. The paper adds that "the reviewed human patches also merged without gold-test failures across the examined pairs." On the most strongly coupled mined pairs, GPT-5.5 showed no interference across 72 runs.
The load-bearing word is validated. To get into that set of 417, a pair had to clear two filters. First, a clean merge: both patches apply to a shared base commit and their composition produces no textual conflict. Second, base validity: each pull request's gold tests must fail on the base with only the test patch applied, and pass once the code patch is added.
Now look at what those filters threw out. In what the paper calls a representative batch, roughly 36% of candidate pairs were dropped because the two changes textually conflicted, another 33% failed base validity, 17% would not apply on a common base, and 14% were accepted.
Roughly a third of the candidate pairs were removed at the mining stage for textual conflict - by tooling every team on earth already runs. The 417 are what was left after taking out everything git would have caught by itself.
So the comforting number does not say parallel agents rarely collide. It says that among changes git merges silently, in a codebase whose history has already been through human review, semantic breakage is rare.
The review is the whole problem, and again it is the authors' point rather than mine. A merged pull request is by construction a change that survived somebody reading it. The population is selected on the outcome. Measuring how often reviewed code breaks reviewed code and calling it a parallel-development rate is like measuring how often insured drivers crash and calling it a road-safety statistic.
One more absence worth noting: there is no concurrency window anywhere in the mining criteria. Nothing requires the two pull requests to have been open at the same time. "Parallel" in that tier means composable from a shared base, not written simultaneously.
Why I believe any of it
Because the small number nearly did not exist, and the reason it does is the best signal in the paper.
The authors caught their own grading manufacturing interference, two ways. Agents sometimes edited test files, which could stop the gold tests being applied consistently. And individual patches were tested only against their own task's tests while merged patches were tested against both - so a patch that broke another task's test all on its own would surface that failure only in the merged run, and get scored as interference. Both bugs, the paper says, "falsely increased" the interference count. After removing the agent test edits and running the combined test set in every condition, "nearly all apparent interference disappeared."
People optimizing for a frightening headline do not go back and find that. It cuts directly against their own interest. These are academics publishing a benchmark, and a new benchmark's incentive is to be alarming enough to get cited. Alongside that, for completeness: the work is supported by NSF grants and by corporate funding and/or compute from Amazon, IntellectAI, Infosys, Tidalwave, Veris, Shopify, Microsoft, Thinking Machines, Dandy, Perplexity and Daytona. None of those interests point toward publishing a downward correction either. They published it anyway.
Two things I could not establish, which belong here rather than in a footnote.
The paper does not say what the interference count was before the fix. We could not confirm it from the text, the tables, the figure captions or the appendices. "Nearly all" gives the direction and nothing else. Anyone who wants the magnitude would have to compute it from the uncorrected result set, which the authors did ship in the repository next to the corrected one.
And we could not verify how the results break down by model. The paper says it evaluates "more than 4,000 graded task solves using two model families," and names two harnesses, mini-swe-agent and OpenHands. GPT-5.5 is the only model named in the results text we could read, and there is no per-model breakdown. The repository makes this murkier rather than clearer: its README says the OpenHands data was computed on the uncorrected metric and is not used for any paper claim. So "two model families" is stated rather than shown, and anyone wanting to know whether this mechanism holds outside GPT-5.5 cannot currently find out from what has been released.
The fix, and the one word that undoes it
Tell the second agent what the first one changed and 89 of 108 constructed runs recover - 82%, rising to 93% among message-conditioned patches that merged cleanly.
A handoff message that recovers four-fifths of the failures is what will be sold as a product, under some name like agent context sharing. Read the provenance first. The paper calls the input the oracle message and is plain about where it comes from: "We generate the message from the scripted transformation, so the result assumes a complete and accurate description of the change." The threats section shuts the door: "The communication results assume complete descriptions of finalized changes; recovery with messages generated during parallel work remains untested."
The message is template-generated from the known answer. A complete, accurate description of a finalized change is precisely the one artifact that does not exist while two agents are mid-flight - if you had it, you would not have the problem.
So the 82% is not a feature anybody can ship. It is evidence about the shape of the failure: the second agent needed a fact, not an architecture. Encouraging. Not purchasable.
Pairs are not fleets
All of this is pair-level, and the paper flags the gap rather than leaving it for a critic: "With N>2 tasks, a failure may require three or more changes to occur together, even if every pair works correctly. Larger task sets also create more possible dependencies."
Whether pair results predict reliability for larger groups is, in the authors' own words, something extending the benchmark "would test." Future work. Not a result. If a vendor extrapolates from two agents to twelve, they have gone somewhere the evidence does not.
The nearest thing anyone measured in production
A separate paper posted three days earlier took a different multi-agent promise into a named industrial setting. Jansson, Strandberg, Sörensen, Enoiu and Afzal built a root-cause analysis workflow for nightly test failures at Westermo Network Technologies AB twice - once as a single agent, once orchestrated - running GPT-5.4, and took operational measurements across 120 repeated executions on two real failure scenarios, with six practitioners assessing the output.
Their summary line: "The multi-agent configuration provided no quality advantage but was approximately three times slower and twice as costly."
Do not read those two halves as one finding. The speed and cost figures are operational measurement across 120 executions. The quality finding is six practitioners' perception, and it is a crossover rather than a clean loss - the paper's own wording is that "this crossover indicates no consistent practitioner-perceived advantage for either architecture." The authors also caution that "the multi-agent implementation should consequently be interpreted as a controlled variant rather than an optimized architecture," and that the benefits of agent architectures "require further evaluation in more complex scenarios."
Different task, different failure mode, small sample, self-evaluated build. Not corroboration. Just the nearest thing anybody has actually measured in production, which tells you how thin the evidence base under this entire product category currently is.
Hype-o-Meter: the parallel-agent coordination problem
Mechanism: real. Demonstrated on 12 real helpers, controlled for the obvious objection in the tier where it could be, honestly caveated.
Frequency: unmeasured. Not "low." Not "high." Unmeasured, by the authors' own statement, pending execution records that do not exist.
Tooling category: premature. Anyone pricing off tasks constructed to fail is pricing off tasks constructed to fail. Anyone relaxing because of the 1-in-417 is relaxing about reviewed human code.
What to actually do
The honest summary of stale is that it established a mechanism and declined to estimate a frequency. Anyone quoting either number as a rate is quoting past the paper.
The method, though, is portable, and it is the thing to take. Take merged pull-request pairs from the repository in question that touched a shared source file, apply both patches to a shared base commit, and run the same combined test suite in all three conditions - A alone, B alone, merged - counting only the tests that fail merged while passing in both solo runs. Using a different test set per condition is the bug the authors had to correct, and it inflated their first pass, so that detail is the method rather than a footnote to it.
Then split the result in two, because the two halves have different answers. If most of your candidate pairs drop out for textual conflict, as about a third of Django's did, git is already catching that part and no tooling budget is required for it. The residual semantic rate - what is left after git - is the only thing a coordination product would actually be selling you.
And nobody should budget for the handoff-message fix on the strength of the 82%. That message was generated from the finished change, and the paper says recovery from messages written during parallel work has not been tested.
One warning for anyone planning to start from the authors' own code, because the paper says "the code is available in the benchmark repository" and that is not quite what is there. At github.com/illinoisdata/STALE-bench the repository root holds a README, a .gitignore, and the instances, patches and results directories. The 417 validated instances, the gold patches, and both the corrected and uncorrected result sets are public and genuinely reusable today. The runner modules the README tells you to invoke are not in the tree. A team can reuse the dataset this afternoon, but it cannot rerun the experiment.
That gap is a small, tidy example of the habit this whole column is about: a claim stated in a paper, and an artifact behind it that does not yet back the claim up. We found it the way the paper recommends finding anything - by going and looking, instead of reading the abstract.
Sources
- Xia, H., Wu, E., Park, Y. Passes Alone, Fails Together: Benchmarking Semantic Coordination in Parallel LLM-Agent Development. arXiv:2609.25396, submitted 21 Sep 2026; 6 pages, accepted to the 2nd Workshop on Explainable and Reliable Software Systems (EXPRESS 2026). Abstract - full text
- Benchmark repository (data and results; see the note above on what it does not contain): github.com/illinoisdata/STALE-bench
- Jansson, E., Strandberg, P., Sörensen, T., Enoiu, E. P., Afzal, W. Supporting Industrial Test-Failure Analysis with LLM-Based Systems: An Experience Report. arXiv:2609.21843, submitted 18 Sep 2026; 16 pages, accepted to PROFES 2026. Abstract
