> ## Content Index > Fetch the complete content index at: https://varops.com/llms.txt > Use this file to discover other available public pages before exploring further. # How to read an AI patch benchmark when a third of its trials forbade the agent from testing - URL: https://varops.com/how-to-read-an-ai-patch-benchmark-when-a-third-of-its-trials-forbade-the-agent-from-testing-2/ - Published: 2026-09-16T14:21:08.000Z - Updated: 2026-09-16T14:21:08.000Z - Description: 1Password's 26% clean-fix rate averaged in trials that forbade the agent from testing and trials told to patch it wrong. Trail of Bits reran the same published data and got 86% - then published its own human failure rate next to it. - Author: Nix Nullty - Tags: Overhyped *The most-quoted number in AI patching is 26%. Trail of Bits reran 1Password's own published data and got 86%. Both firms sell security, so neither figure walks in on trust - but the gap between them is not a scandal*; *it is a method. Today* [***Nix Nullty***](https://varops.com/columnist/nix/) *takes the two benchmarks apart and hands over the three questions that separate a measurement from a restriction. The third one is the useful one, and almost nobody runs it. —* [*Muximus*](https://varops.com/columnist/muximus/) --- More than a third of the trials behind the most-quoted number in AI patching prevented the agent from compiling or running its own code. Another fifth instructed the agent to apply the wrong fix on purpose. All of it went into one average, and that average is 26%. Pull those trials back out and the same dataset says 86%. Nobody faked anything, and that is the point worth staying for. The 26% comes from [1Password's FLAWED report](https://1password.com/blog/why-ai-generated-patches-still-require-human-review?ref=varops.com), published 2026-08-06 by Off-by-1 Labs, which the company describes as its "brand-new security research team." The 86% comes from [Trail of Bits reanalyzing that study's own published data](https://blog.trailofbits.com/2026/09/15/1passwords-ai-patching-benchmark-is-misleading/?ref=varops.com) on 2026-09-15\. Both firms sell security. Neither number gets in here on trust. What is worth an operator's attention is not which one wins. It is that telling them apart takes three questions, and those three questions work on the next benchmark that turns up in a [board deck](https://varops.com/the-adoption-number-in-your-board-deck-is-wrong/). ## Credit where it's due, first, because it is load-bearing 1Password published the code and the data. That is why any of this is checkable. The reanalysis that embarrassed the study exists only because the study was open, and a security vendor that ships its raw material knowing what a competitor might do with it has earned the sentence. Trail of Bits then did the harder version: it published its own failure rate. More on that below, and it is the most interesting number in the story. Hold both of those, because the rest of this column is going to be less generous. ## The claim, stated the way its authors would state it The study is not small and it is not hiding. 6,480 patches generated across six recently-disclosed CVEs, using two frontier reasoning models - ChatGPT 5.5 with Trusted Access for Cyber guardrails and Opus 4.8 with the Cyber Verification Program equivalent - and 6,080 reported on, after the runs where a model was caught going to look up the real upstream fix were dropped. [The code and data are on GitHub](https://github.com/Off-by-1-Labs/FLAWED?ref=varops.com). The six vulnerabilities were picked because their fixes were complex, and the report says so plainly rather than pretending otherwise. Its conclusion - that AI-generated patches still require expert human review - is entirely defensible. 1Password's interest in reaching it is the ordinary kind: a security company arguing that expert review remains necessary is arguing toward its own market. Now the two things that make a careful reader slow down, neither of which either side has an incentive to raise. The small one first. The same blog post says "we produced 6,080 patches" near the top and "all 6,480 patches we generated" near the end. Both are true of different quantities. Only one of them is a denominator. The large one: **the report has two overall numbers, and which one a reader met depended on which document they opened.** The paper's figure is 26.0% - "Across our entire dataset, only 26.0% (S1) of patches fully mitigated the target vulnerability without any ill side effects." The blog post announcing it opens on something else, in bold: "Our data shows that LLMs produce Fix-Like Artifacts with Embedded Defects (FLAWED) 53.9% of the time when complex patches are required." The string 53.9 appears nowhere in the paper. It is an aggregate assembled for the announcement. And when 1Password restated it further down that same post, and again on X the same day, the qualifier - *when complex patches are required* \- came off. So the number that traveled furthest is the one the paper does not contain, minus the condition that made it mean anything. Which is, one layer early, exactly the failure this whole story is about: an average built from parts that measure different things, shipped without the parts. ## What the reanalysis took out Trail of Bits names four design choices. Two are about the sample and the settings: the six bugs were selected for difficulty, with clean-fix rates running from 3% to 60% across them, and the models ran at their tools' defaults - medium effort for ChatGPT 5.5, high for Opus 4.8 - neither at its highest setting, with nobody measuring what changing it would do. The other two move the number. The prompt classes that instruct a wrong fix: 22% of the data. The evaluation mode that stops agents building or running code at all: 36%. That 22% is also where the 53.9% comes from. Dan Guido, Trail of Bits' CEO, made the point on X three days before the reanalysis ran, and it is the cleanest statement of the problem anyone has managed: the 53.9% is "an average over 9 prompt styles, where 2 of the 9 deliberately instruct the prompt to apply an incorrect patch and see if it obeys." Both of the report's overall numbers average in trials built to fail. That is a legitimate experiment and it is not a measure of ordinary repair work, and the two should never have shared a denominator. Keep only the trials where agents could run code and were not told to apply the wrong fix, and 2,634 of 3,067 patches - 86% - blocked the supplied exploit. Before that figure gets forwarded anywhere: Trail of Bits attaches its own limit to it, which is more than most reanalyses bother with. "Blocking that exploit does not establish a complete repair." The 86% and the 26% measure different things - a weaker test on a cleaner subset. Read as *these models are far better than 26% when allowed to work normally*, it holds. Read as *86% of AI patches are clean*, it does not, and Trail of Bits does not claim it. ## The graders, which is where the whole thing gets shaky Everything above is a fight about which trials belong in the denominator, and both sides have a case. The grading is not that. It caps how precisely any figure from this study can be read no matter whose trials get counted, and neither firm's position improves it. The verdicts were assigned by models. The two grader models gave different outcomes on 36.8% of the same patches, and the reported figure averages their opinions. Checked against human reviewers, the automated grades matched on the full five-category outcome 65.9% of the time - 87.7% on the narrow question of whether the original bug got fixed, and 70.5% on whether new bugs were introduced. That last one is the number that matters, because introducing a new bug while closing an old one is the exact failure the study is named after. The instrument is at its worst on the thing it was built to detect. The concrete errors run both ways, which is why they deserve itemizing instead of a summary. Agents given a proof-of-concept exploit were told to stop once their patch defeated it. They were then graded on vulnerable paths the exploit never touched - marked down for not fixing what the instructions told them to stop before. Agents were told to leave existing tests alone, including where a correct fix requires updating a test to reflect deliberately changed behavior. Eight percent of ActiveMQ verdicts penalized an intended behavior change as a regression. Running the other way, the Chromium grader passed patches that left a use-after-free sitting in a callback. And then Linux, which is the one to remember. The upstream reference fix - the human-written patch the agents were being measured against - contains an off-by-one error. 248 generated patches faithfully reproduced it, because reproducing the official fix is what a good patch is supposed to look like. The grader caught the resulting vulnerability in 24 of them. The other 224 went through clean, carrying a real bug, in a study whose entire premise is that models ship fix-like artifacts with defects embedded in them. Two graders that disagree on more than a third of the same patches produce an error bar wider than most of the differences anyone is trying to read off the results. This is the same problem [an agent benchmark has when it scores the harness as much as the model](https://varops.com/a-benchmark-scores-the-harness-as-much-as-the-model-openais-own-number-swings-3x/), and it applies to the 86% exactly as hard as to the 26%. ## The denominator nobody publishes Here is the part that changes what both figures mean, and it is the reason this belongs in front of a leader rather than a security team. Trail of Bits went back through its own consulting records: the first fix a client submitted for each of 2,265 vulnerabilities, across 236 security assessments between 2024 and 2026\. Favorable conditions, all of them. The developers maintained the software themselves. They held a detailed report from the engineers who found the bug. They knew their patch would be reviewed. 283 of those first fixes failed to fully resolve the issue. 12.5%. One in eight, with a 95% confidence interval of 10.5% to 14.5%. Almost every AI patching number in circulation is being read against a human developer who gets it right the first time, and that developer does not exist. Trail of Bits is careful that this is not head-to-head - "a direct comparison with agents would require the same tasks and working conditions" - and flags that its own figure may undercount, because a client told about a flawed patch in an informal conversation fixes it before it ever reaches the formal review record. An anchor, not a verdict. Most people reading a benchmark do not have one at all, which is most of why benchmarks land as hard as they do. ## What happened when the patches met actual maintainers Lab figures are arguments. Merge rates are outcomes, and a merge rate is what an operator can act on Monday. Through Patch the Planet, a joint initiative with OpenAI, Trail of Bits has been sending agent-written patches - engineers directing the work and checking the results - to widely used open-source projects. Of the 186 pull requests maintainers had merged or closed by 2026-09-14, maintainers merged 126\. An acceptance rate of 67.7%. In 91 of those 126, or 72.2%, the security fix went in as originally proposed with no security-relevant revision observed. Another 33 were accepted after one. The obvious caveat is Trail of Bits' own - "maintainer acceptance does not establish that every patch is correct" - and it is still the most useful figure in the entire dispute. A grader model has an opinion. A maintainer merging a patch into their own project is expressing a preference about code they will have to live with. The 60 that closed without merging read worse in the raw count than in the breakdown: 36 superseded or re-landed elsewhere, 14 closed for policy, process, scope or maintenance reasons, 3 duplicates, and 3 the report could not place. Four were rejected on technical grounds. Then the part most firms would have left in a drawer. Trail of Bits went through roughly 33,500 subsequent commits in those projects, hunting for changes that repaired something its own patches had broken. The sweep found at least ten functional bugs, four build, test or release automation bugs, and one performance bug. It found no exploitable security vulnerabilities. Ten functional bugs is not a clean sheet. The honest shape of that result is better evidence than a clean sheet would have been, and anyone who has read enough vendor case studies knows exactly why. Two of the tools behind this work are now downloadable, which is rarer than it should be. [Post-patch-validation](https://github.com/trailofbits/skills/tree/main/plugins/post-patch-validation?ref=varops.com) walks an agent through reproducing the original bug with a test that fails before the patch and passes after, testing a second path to the same failure, and treating a broken test run as inconclusive rather than as evidence of a fix. [Review-walkthrough](https://github.com/trailofbits/skills/tree/main/plugins/review-walkthrough?ref=varops.com) is for the engineer doing the merging. Both ship as plugins in Trail of Bits' own marketplace. One caveat belongs on the first, and to its credit Trail of Bits supplies it rather than letting the reader assume: post-patch-validation is new and "was not used in the Patch the Planet work described above." The 67.7% was achieved without it. ## The freenginx case, which settles more than either percentage 1Password's report includes a case study of a Patch the Planet fix for a memory-safety bug in freenginx's embedded Perl module. The agent-written patch left one vulnerable path open and introduced a new crash during request cleanup. Trail of Bits' answer to that section is one sentence: "The paper's criticism of our patch is correct." What the report also documents, and what is very easy to skim past, is what happened next. The maintainer closed the pull request and wrote his own fix. It covered all three vulnerable paths. It introduced the same crash during cleanup. Both fixes kept a callback alive so freenginx could use it later. If the request timed out first, freenginx would make the request unusable and then release the callback - and releasing it could run Perl code still reaching for a request that no longer worked. Catching that meant looking past the original bug to what happens when a request ends early. Neither author did. "Two authors, one human and one agent, working separately, made the same mistake on the same bug." That sentence is doing more work than either headline percentage, and it is sitting inside the report that the percentages came from. ## Both interests, and the fight that predates the blog post Trail of Bits is a security consultancy that audits code, partners with OpenAI on the initiative the report criticized, and shipped two agent plugins the same day it published the reanalysis. Its CEO is a co-author. That is not a disqualification. It is also not nothing, and a reader who weighs the 1Password interest without weighing this one is only doing half the job. The disagreement did not start with the blog post either. Guido spent 2026-09-12 arguing about the study on X with its authors, three days before the reanalysis appeared, and called it "based on only 6 cherry picked patches" - his phrasing, and imprecise, since the six were CVEs rather than patches. Keith Hoodlet, who heads Off-by-1 Labs at 1Password, answered the conflict-of-interest implication head on: "I designed this study two months prior to the announcement of Patch the Planet, and had no foreknowledge of OpenAI's relationship with Trail of Bits." A separate critique of the report by Davi Ottenheimer had already been [published on 2026-09-07](https://www.flyingpenguin.com/disinformation-pushed-by-1password-ai-patching-report-is-false/?ref=varops.com), before either of them posted. As of 2026-09-16, 1Password has not publicly responded to the reanalysis. The blog post carries no correction or update and its modified timestamp still reads 2026-08-06, the FLAWED repository has had no commit since 2026-08-04, and both researchers who argued the case on X went quiet on 2026-09-12\. That is a VarOps check rather than a sourced claim, and it could be overtaken by the time this is read - the study's authors have already demonstrated they will defend it. ## What we could not check We did not rerun either dataset. The 26.0% and the 86% are both reported figures from parties with something to gain, one recomputed from the other's data. The 12.5% baseline is Trail of Bits' own consulting record, self-described as possibly undercounting, and by its own authors not a like-for-like comparison. Nobody - not the report, not the reanalysis - measured whether running the models at their highest reasoning setting would have changed the result. And the archive service that would have shown whether the 1Password post was quietly edited was down when we looked, so the unchanged finding rests on the page's own timestamps. ## The three questions Ten minutes, on any benchmark that [arrives with a number attached](https://varops.com/how-to-check-an-agent-time-savings-claim-when-the-slowest-step-is-not-software/). **Could the agents build and run their code?** A trial that forbids compiling and testing is [measuring the restriction, not the model](https://varops.com/an-agent-benchmark-measures-the-score-not-the-capability-heres-the-audit-that-put-a-number-on-the-gap/). If a benchmark pools restricted and unrestricted trials into one average, that average means nothing until the split is published. **Are adversarial trials in the denominator?** Prompts that deliberately instruct a wrong fix are a fine experiment and a terrible ingredient in a headline rate. Trail of Bits' own list of what a useful patching benchmark should measure makes "report misleading prompts and restricted tool access separately" one of its five principles - a standard worth holding it to as well, next time it publishes a number of its own. **What is the human baseline?** 12.5% of first fixes, from motivated developers working on their own code with a detailed bug report in hand and a review waiting, did not resolve the issue. Any AI figure read against an implied zero is being read wrong. If you have ever seen a vendor number presented next to nothing at all, that is the move - and it works because the comparison is left for the reader to supply. If only one of those gets run this quarter, make it the third, and point it inward rather than outward. Before the next vendor number lands, pull the team's own record of security fixes that needed a second pass. Most ticket systems will produce it in an afternoon. It is the only baseline that describes the actual organization, and once it exists the next benchmark arrives as a comparison instead of a scare. The freenginx maintainer had every advantage available - his own codebase, his own timeline, no experiment watching him work - and made the same mistake the agent made. A benchmark that cannot see that is not measuring patching. It is measuring the conditions it imposed.