> ## Content Index
> Fetch the complete content index at: https://varops.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# An open agent beat OpenAI’s security product. The edge was a text file, not a model.
- URL: https://varops.com/an-open-agent-beat-openais-security-product-the-edge-was-a-text-file-not-a-model/
- Published: 2026-06-25T08:58:18.000Z
- Updated: 2026-06-25T08:58:18.000Z
- Description: An open-source agent beat OpenAI's Codex Security on the researchers' own benchmark, 11.3% to 9.2%. The edge wasn't a bigger model - it was a $1,400 text file. The receipts, and the fine print.
- Author: Rex Factor
- Tags: Proof of Work

*Every vendor selling you an AI security tool is, underneath the pitch, selling you their model. So here is an awkward result for the procurement deck: what if the model was never the moat? A research team froze the model and the harness and let a plain-text playbook teach itself, and the cheap, open setup beat OpenAI’s commercial product on the authors’ own benchmark.* [***Rex Factor***](https://varops.com/columnist/rex/) *does what Rex does. He lays the receipts on the bench, then reads you the fine print the headline is about to lose. —* [*Muximus*](https://varops.com/columnist/muximus/)

Start with the outcome, because it is the part that should change a buying conversation. An AI agent running an open-source model found real vulnerabilities at a higher rate than OpenAI’s commercial [Codex Security](https://www.helpnetsecurity.com/2026/06/23/codex-security-ai-security-auditing/?ref=varops.com) product: 11.3% against 9.2% on the researchers’ [target-match metric](https://arxiv.org/abs/2606.16420?ref=varops.com). Before anyone reaches for that as a banner, note two things the researchers note themselves, up front: it is their own measurement, and the baseline is one they openly call imperfect. We will get to the fine print. But the headline is real, and the reason behind it is the interesting part. The open agent did not win on a bigger model. It won on a plain-text document it wrote for itself.

The system is called EvoHunt, from a paper titled [“Transferable Self-Evolving Playbooks for Agentic Security Auditing”](https://arxiv.org/abs/2606.16420?ref=varops.com) (Ziyue Wang and five co-authors, posted to arXiv on 15 June 2026). The design is the whole point, so it is worth slowing down on. An AI auditing agent is three parts: the model that reads the code, the harness that lets it navigate and run tools, and a playbook of procedural knowledge that tells it how to hunt. The usual way to make such an agent better is to swap in a bigger model, a new harness, and a fresh workflow all at once, then hand the credit to the model. EvoHunt refuses that. It bolts the model and harness in place and lets only the playbook change. Whatever improves, you know exactly what improved.

## How the playbook teaches itself

The mechanism is a loop of three agents. One audits a codebase and reports what it finds. A second scores those findings against known answers. A third rewrites the playbook from the failures. The playbook [starts as an empty file](https://www.helpnetsecurity.com/2026/06/23/codex-security-ai-security-auditing/?ref=varops.com) and grows with every accepted edit, each version committed like code in a git repository. The two playbooks the team grew this way landed at roughly 1,600 and 2,200 lines of procedure the agent wrote for itself.

The test is built to stop the agent from memorizing its way to a good score. Cases come from the GitHub Advisory Database, [split by date](https://www.helpnetsecurity.com/2026/06/23/codex-security-ai-security-auditing/?ref=varops.com): the agent learns on bugs disclosed from 2023 through 2025, then is graded on bugs disclosed in 2026 that it has never seen. Every case runs in a sandbox, and the team kept only serious bugs an outside attacker could actually reach. That is a cleaner setup than most vendor demos you will be shown this year.

## By the numbers

The figures arrive in two grades, and an evidence desk keeps them apart. The benchmark numbers are measured in the paper. The cost and the field tallies come from a Help Net Security interview with the authors. None of it has been reproduced by an independent third party yet.

- Open agent vs commercial product: an OpenCode harness running the open-source GLM5.1 model cleared Codex Security on [every metric the team tracked](https://arxiv.org/abs/2606.16420?ref=varops.com), 11.3% to 9.2%, across [371 test cases](https://www.helpnetsecurity.com/2026/06/23/codex-security-ai-security-auditing/?ref=varops.com).
- Same method on a closed stack: adding an evolved playbook to Codex running GPT5.4-xhigh raised end-to-end exploits [about sixfold, 1.1% to 6.2%](https://arxiv.org/abs/2606.16420?ref=varops.com).
- Transfer to cheaper models: the GLM-grown playbook lifted [Qwen3.6-27B from 2.4% to 6.5%, and Qwen3.6-35B-A3B from 1.1% to 4.6%](https://arxiv.org/abs/2606.16420?ref=varops.com).
- Cost of the teaching run: [about $1,400](https://www.helpnetsecurity.com/2026/06/23/codex-security-ai-security-auditing/?ref=varops.com) over one month on subscription accounts, after which a small Qwen model recovered most of the performance at roughly a third of the cost per case.
- Field results, not just a benchmark: [28 maintainer-confirmed zero-day disclosures across 18 open-source projects](https://www.helpnetsecurity.com/2026/06/23/codex-security-ai-security-auditing/?ref=varops.com), plus one $1,500 bounty, with six more confirmed since the paper went up.

The line that matters most to anyone signing a contract is transfer. A playbook grown by an expensive teacher made cheaper, weaker models meaningfully better at the same job. The expertise lives in a text file any compatible model can pick up. Pay for the costly teaching step once, then run the result on inexpensive models for as long as you like. That is not a model advantage. That is a procedure, and procedures copy.

## The two audit personalities the loop invented

Left to evolve on their own, the two playbooks settled into opposite working styles, and the contrast lands on a call every security team makes. The GPT-grown playbook became a precision instrument: it limits how many bug types it chases at once and refuses to report anything without a working, reproduced exploit. In a sampled set of its findings, [none were false alarms](https://www.helpnetsecurity.com/2026/06/23/codex-security-ai-security-auditing/?ref=varops.com). The GLM-grown playbook went the other way, an exhaustive sweeper that repeats the order to keep looking more than thirty times and accepts thinner evidence to widen the net. It caught more and left more for a human to triage. A short list you can trust against a long list you have to sort. The loop rediscovered the oldest tradeoff in the work and wrote it down in plain text.

## The fine print, because it is the story too

Now the part the aggregators will drop, and the part Rex cares about most. The authors flag their own limits rather than bury them, which is itself worth crediting. Codex Security is a [separate product](https://www.helpnetsecurity.com/2026/06/23/codex-security-ai-security-auditing/?ref=varops.com) running a different model and harness, so this was never an identical-agent baseline. The clean test would pit an evolved playbook against an expert-written one inside the very same agent, which the team says is hard to arrange because the best expert workflows are proprietary. Match rates sit in single digits across most runs, so nobody here is finding most of the bugs. Each playbook grew in a single training pass, which means luck could account for some of the gap. And the judge that scores the findings is itself an AI model. Co-author Wang ties the work to [Rich Sutton’s “Bitter Lesson”](https://www.helpnetsecurity.com/2026/06/23/codex-security-ai-security-auditing/?ref=varops.com), that general, scalable methods beat hand-encoded domain knowledge, and calls it a “0 to 1” step, not a finished product. Take him at his word on both halves of that.

So the honest takeaway is narrower than “open source beat OpenAI,” and more useful. The defensible asset in agentic security tooling may not be the model a vendor licenses you. It may be the procedure, and procedure is cheap to grow, cheap to run, and portable across models. The next time you are quoted a premium for an AI auditing product, you have a sharper question to ask than you had last week: what, exactly, am I paying for that a $1,400 text file can’t be taught to do?