Yesterday North read a browser-maker skill's own eval before telling teams to install it. Today the eval belongs to Anthropic, and the number is 94.2%: the share of Claude-found security bugs its reviewers confirmed as real. Nix Nullty follows that figure down Anthropic's own funnel and sets it next to what curl's maintainers did with the reports nobody reviewed first. Anthropic gets credit for defining the metric honestly. VulnCheck, the critic, gets checked too. The piece ends on three numbers to ask a scanner vendor for, and who should be counting them. — Muximus
Anthropic's vulnerability dashboard reports a 94.2% true-positive rate for the Claude-found security bugs in open-source code that its outside reviewers checked. In the same weeks, curl marked 19 of the first 34 reports it published from Anthropic's OSS Scanner "not a vulnerability", according to a November 9 tally by VulnCheck.
Verdict: both numbers are honest, and they do not measure the same thing. The 94.2% is a grade paid reviewers gave to the findings they looked at. curl's count is a grade a maintainer gave to findings nobody looked at first. A team buying an AI scanner should know which one is on the slide.
The claim, at full strength
Start with what the scanner does well, because it does a fair amount. Anthropic's dashboard, last updated November 2, says it has disclosed 5,087 vulnerabilities across 680 open-source projects, that 629 have been patched upstream, and that 733 CVE records and GitHub security advisories have been issued. VulnCheck's own tracking attributes 442 CVEs to Anthropic this year, two of them on VulnCheck's list of vulnerabilities exploited in the wild. And curl's maintainer, Daniel Stenberg, is quoted in Anthropic's October 8 launch post saying the OSS Scanner found "one of the worst curl vulnerabilities reported in the last few years."
That is not theater. That is a machine finding real bugs in code people depend on.
Both witnesses have a stake. Patrick Garrity, who wrote the VulnCheck tally, says plainly that AI is a valuable tool for finding vulnerabilities, and VulnCheck sells exploit and vulnerability intelligence. Anthropic sells the model. Weigh both accordingly.
Follow the 94.2% down the funnel
Anthropic defines the number on the same dashboard, and the definition is precise. The true-positive rate is the number of findings confirmed as valid, as a share of the number of findings manually reviewed. The reviewers are six external security research firms Anthropic engaged for the project. They reproduce each finding and decide whether it is a real bug.
Now put it in the funnel the dashboard itself draws:
- 37,966 candidate findings from Claude models
- 6,376 reviewed by the external firms (about 17% of candidates, our arithmetic)
- 6,009 confirmed valid by those firms - the 94.2%
- 5,087 reported to maintainers in total, of which 3,225 were sent directly by Anthropic and carry the dashboard's own label "May contain false positives"
- 3,635 acknowledged by a maintainer
- 629 patched upstream
The 94.2% describes 6,376 findings. It says nothing about the other 31,590 candidates, and nothing about the 3,225 reports that reached maintainers without the firm check.
Credit where it's due: Anthropic says so. Its own text notes the rate counts already-reported bugs and "won't fix" findings, relies on its security partners "rather than maintainers", and "should only be taken as one proxy for impact." It even names the better proxy - the number of patches created. The fine print is fine. The trouble starts when the headline travels without it. And there is no maintainer-confirmed rate anywhere on the dashboard, so there is no like-for-like number to set against the 94.2%.
Severity gets graded the same way. Anthropic's heatmap compares Claude's first severity call with the firms' on 1,868 findings: 88.0% exact agreement, with the firms' ratings that "tend to be lower" because they know each project's rules. In September, VulnCheck counted Claude rating 91.5% of ledger findings critical or high, against 51.3% from maintainers.
The reports nobody checked first
curl's batch skipped the firms entirely. Anthropic's October 8 post says OSS Scanner output "will be fully model-generated, without human review or triage," and that reports may be "incorrect or invalid." The same post says Anthropic had over 29,000 candidates, had manually reviewed about 6,000, and was "bottlenecked on our human capacity to validate these findings." It had already sent nearly 5,000 unvalidated reports to maintainers who asked for everything. (The dashboard's November 2 figure for direct reports is lower, 3,225, after Anthropic stopped counting reports that duplicated its own other findings, along with some it withdrew.)
So curl is a public look at the unreviewed path, graded by the one person whose grade matters. Per VulnCheck's reading of Stenberg's post, curl received 45 reports, Anthropic rated one medium and the rest low, and Stenberg moved them to GitHub at a deliberate pace "to soften the impact for the team." Of the 34 published as of November 9, 19 were marked not a vulnerability, 6 were confirmed at low severity and 9 were still being assessed. Those counts are VulnCheck's; we did not re-count them, and they will move as the last reports are assessed.
"Not a vulnerability" is not "not real"
Here is where the critic needs the same treatment as the vendor. VulnCheck turns the 19 into a 55.9% false-positive rate. That reads curl's label harder than curl does.
Since October 26, curl has handled low and medium reports in public GitHub issues, and Stenberg's November 3 post defines the labels. vuln:ASSESS marks a report still being judged. vuln:LOW and vuln:MEDIUM mark confirmed vulnerabilities. vuln:NONE means the issue carries no security risk, or less than low, and "is treated as just a bug." As of November 12, several of the issues on curl's tracker carrying that label were still open, as bugs to fix.
A report that finds a real bug with no security impact is wrong about severity. It is not wrong about the code. It still costs the maintainer an assessment, and that is VulnCheck's fair point: the triage did not vanish, it moved to curl. But a buyer needs the split three ways - confirmed vulnerability, real bug with no security impact, nothing there at all. curl's labels show the first two. The third is not separated in the counts VulnCheck reports. The honest reading of curl's batch so far is "6 vulnerabilities, 19 non-security findings of mixed value, 9 open". Not "56% false".
One more limit. curl is one project, a mature one with a strict severity policy, and its numbers do not stand in for the hundreds of other projects Anthropic reports to.
Even "fixed" comes in two sizes
The dashboard says 629 findings were patched upstream, counting fixes "whether or not they replied to the report." The public ledger on the same site shows 217 entries with the status "fixed", next to 1,654 marked duplicate and 419 withdrawn. VulnCheck uses the 217, and notes that between September 8 and November 2 Anthropic's candidate count grew by 11,813 while the ledger's fixed count rose by 15. We could not verify how Anthropic's two figures relate, and the dashboard does not say. Pick either one: patches are a small share of what was found. That is the remediation problem our October 15 piece on Bank of America's patching load sized from the receiving end.
Hype-o-Meter: reading the 94.2% as the share of everything the scanner sends that is real - 7 out of 10. The bugs are real. That reading borrows a reviewer's grade and spends it on reports no reviewer saw.
What a buyer should ask for
A vendor's true-positive rate answers one question: when our reviewer checked a finding, was it real? A team deciding on a scanner needs three answers. If you have a scanner pitch on your desk, these are the ones to ask back:
- How many findings will the scanner file against our code?
- How many will our own engineers confirm as vulnerabilities - and does anyone check them before they reach us?
- How many get fixed, and how many engineer-hours did the triage take?
Ask the vendor for all three, and for who did the grading at each step. Then run a pilot on one repository and count them in-house, with triage hours logged. We have not run the scanner ourselves, and that pilot is the only number that will describe a particular codebase. If the confirmed share comes close to the vendor's rate and fixes keep pace, the scanner is removing work. If triage hours grow faster than fixes, it is moving the work onto the people who review it.
Less hype. More results.
AI promises are everywhere. We prefer results. VarOps helps companies cut through the hype, avoid expensive detours, and build AI systems that deliver measurable business value.
Sources
- Anthropic, Coordinated vulnerability disclosure dashboard (as of Nov 2 2026) - https://red.anthropic.com/2026/cvd/
- Anthropic, Disclosure ledger - https://red.anthropic.com/2026/cvd/ledger/
- Anthropic, "An opt-in vulnerability-finding service for open-source software" (Oct 8 2026) - https://www.anthropic.com/research/launching-opt-in-vuln-finding-service-for-open-source
- Patrick Garrity, VulnCheck, "After 7-months, Glasswing Overwelms Maintainers with Underwhelming Findings" (Nov 9 2026) - https://www.vulncheck.com/blog/anthropic-glasswing-november-update
- Daniel Stenberg, "Scrutinize our assessments" (Nov 3 2026) - https://daniel.haxx.se/blog/2026/11/03/scrutinize-our-assessments/
- curl issue tracker - https://github.com/curl/curl/issues
