Berkeley ran the same model through three coding agents and got the same work out of all of them, at up to five times the price. North’s interest is not which one won - it is that most budgets treat “which coding agent” as a taste question and “which model” as the money question, and the measurements say that is backward. She lands on a test any team can run in an afternoon, and on one reason it might go wrong. - Muximus
Every coding agent is two purchases wearing one name. There is the model, which determines whether the work gets done. And there is the harness - the software that briefs the model, hands it its tools, and manages the job - which determines what getting it done costs.
Most budgets price the first one. The second arrives by default, chosen by whoever installed something first, and it is where a measurable share of the money goes.
Here is the shape of it, from a study published this week. Claude Fable 5 solved 97.8% of its attempts running inside Claude Code and 96.7% running inside Pi, a minimal open-source harness with four tools. It took 15.3 turns in the first and 15.4 in the second. It cost $1.33 per rollout in the first and $0.67 in the second.
Same model. Same tasks. Same number of turns. Twice the money.
What was actually measured
HarnessTax comes out of UC Berkeley’s Sky Computing Lab - Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Ion Stoica, and Matei Zaharia, with Wei-Lin Chiang of Arena. It tested 21 model-harness pairs: seven models across three harnesses - Claude Code, Codex CLI, and Pi - on two public benchmarks, SWE-bench Lite and Terminal-Bench 2.0.
The method deserves a paragraph, because a number without its method is a marketing claim. The same 30 randomly sampled tasks from each benchmark, three runs per task per pair, each harness started from its own factory settings at its highest effort level, attempts capped at 100 turns, and success scored by each benchmark’s official grader rather than by the authors. Costs come from a fixed price list dated September 1, 2026, applied identically to every model across all three harnesses.
One setup choice matters more than the rest, and the authors state it plainly: for the SWE-bench Lite runs they cut off internet access, disabled the default web tools in Claude Code and Codex, and refused hosted tool declarations at the API level. So this compares harnesses with one feature switched off. That is a defensible way to isolate a single variable, and it is also a real boundary on how far the result travels. Better said before the numbers than after them.
The result splits cleanly in two. Across shared models, using geometric means of cost ratios, Claude Code costs about 2.0x Pi and 1.6x Codex on SWE-bench Lite, and about 1.5x Pi on Terminal-Bench 2.0. The study’s own framing of the ceiling is that the same model can reach similar success rates at up to 5x the cost. Meanwhile, the average effect of harness choice on whether the task succeeded stays within about two percentage points on SWE-bench Lite and about five on Terminal-Bench 2.0.
One number moved a lot. The other barely moved at all.
Now the interests, because this is a study whose conclusion flatters the cheap option and a reader is entitled to weigh that. Arena sponsored the API access for the test runs and one co-author works there. The Laude Institute supplied Anthropic API credits, the Amazon AI Fellowship supplied compute credits, and Sky Lab’s standing corporate gift list includes Google, IBM, Intel, AMD and SAP among a dozen names. Anthropic credits underwriting a finding that makes Claude Code look expensive is worth saying out loud rather than hinting at. The phrase “harness tax” is not Berkeley’s either - the page credits it to a Portkey post from April, and Portkey sells an AI gateway.
The meter starts before the work does
The money is spent at the front of the job. Every harness opens by sending the model a standing brief - instructions on how to behave, plus a description of every tool it may use - and the model is billed for reading it. Across all seven models, Claude Code’s opening brief runs more than ten times the size of Pi’s, alongside longer instructions and larger tool descriptions. That figure includes the task itself, so it is not purely the harness’s doing, but the ratio between the two is.
The turn counts are what make this a pricing story rather than a quality story. If Claude Code had cost twice as much because it worked twice as long, the answer would be “it is more thorough” and there would be nothing to discuss. It took 15.3 turns against Pi’s 15.4 - fractionally fewer - for a gain of about one percentage point in success, at roughly double the price. The authors put it as higher spending per recorded turn, with the caveat that harnesses count turns differently.
Those are two separate numbers and they are worth keeping apart, because each is cheap to read at a different moment. The size of the opening brief is visible on the very first exchange. The per-turn spending is visible after one completed job. Neither requires a quarter of invoices to find.
Nathan Sutton measured the same thing from the opposite end, putting nine harnesses through a local model on a MacBook Pro - one shared server, a proxy forcing identical settings on every harness, eight small programming tasks each. His figures come from the server’s own accounting rather than from what each harness reports about itself, which is the detail that makes his table usable. The opening brief ranges from 1,171 tokens for mini-swe-agent and 2,008 for Pi up to 16,263 for crush and 18,046 for opencode. In waiting time on that laptop: 21.6 seconds before the first word appeared with Pi, 225.7 seconds with opencode.
He also discloses that he maintains chad, one of the nine harnesses in his own table, and that it posts the top score in it at 24 of 24. Read the comparison accordingly; his raw records are committed in the repo for anyone who wants to recompute them.
Four tools, and the conclusion not to draw
Pi reaches the best available cost-for-results tradeoff on both Berkeley benchmarks while offering the model four tools: read, write, edit, run a command. That is the uncomfortable part and it should not be softened. A great deal of tool surface is not buying the results it is charged for.
The tempting next move is to go count tools and cut some, and Sutton’s table says that is the wrong lever. Sort his nine harnesses by the size of their opening brief and the tool counts scatter: opencode offers 10 tools and carries the heaviest brief in the set at 18,046 tokens, while cline offers 26 and carries 5,876. Codex offers 10 and carries 7,804. The weight is in how verbosely the instructions and tool descriptions are written, not in how many tools there are.
And leanest is not best. mini-swe-agent has the smallest opening brief of the nine and the worst results - 11 of 24, with 14 runs timing out. Trimming is not a strategy on its own.
The objection, which is a real one
Here is the strongest case against reading any of this as a tax, made on the Hacker News thread by a commenter posting as jswelker: much of the extra weight in Claude Code and Codex is ostensibly there for security and alignment, and “leaving those dimensions out and calling it a tax is disingenuous, just turning insecurity into a negative externality.” His analogy is that dumping in the ocean is cheaper than paying for waste disposal.
That deserves a straight answer rather than a wave. The answer is that the two costs are separable. Another commenter, roywiggins, replied that it is not hard to sandbox Pi “without adding anything substantial to the actual size of the system prompt, which is what HarnessTax is mostly measuring,” and reckoned a wrapper costs approximately zero tokens - his estimate, not a measurement. A third made the architectural version of the point: containment belongs in the plumbing around the agent, not in the instructions given to it, because instructions are not enforcement.
Containment is a capital line. It does not have to be a per-transaction one.
But the objection leaves something standing and it should be said. Nothing in any of these sources measures whether Claude Code’s or Codex’s additional instructions actually prevent anything. What the weight buys is an open question, and roywiggins volunteered the flip side of it too - that Pi shipping with no built-in sandboxing at all is a gap rather than a feature. Nobody here has settled this. A measured cost sitting next to an unmeasured benefit is an uncomfortable place to decide from, and it is an honest description of where the evidence currently is.
Sutton’s own account of why the weight exists is the fairest reading anyone has offered: “Opencode’s 18k system prompt is there because it helps frontier models deployed behind an API. The 26 tool schemas in crush are fine when you have 200k context and prefill is instantaneous. All of these choices were made in an environment where the prefill is virtually free.”
The overhead is not carelessness. It is a sound design for one operating environment, being billed for in another.
Which bill it lands on depends on the working space
A second study gets at that distinction between operating environments, and it is the one that explains why the first study’s numbers look the way they do.
A preprint posted in August by Sydney Lewis - arXiv:2608.26218, single author, not peer reviewed, artifacts published - compares two configurations of the same harness rather than two different harnesses. One feeds the model the full conversation in order. The other keeps the same record but automatically shortens older material as the working space fills, and intervenes when the agent starts repeating itself.
Under a tight working space the difference is not subtle. On 169 tasks with a 20,480-token limit and a hard four-and-a-half-minute cutoff, the second configuration raised the share of tests turned from failing to passing from 28% to 49%, and complete solutions from 43 to 72 - counts, not percentages. The same unmodified configuration lifted both figures for three additional models of different designs, with no retuning for any of them.
Widen the working space and the effect mostly goes quiet. In the paper’s wide-window comparisons, run on Qwen3.6, the two configurations land close on two of the three benchmarks. Mostly, not entirely: on the third, FeatureBench, the managed configuration still held a lead even with room to spare. So this is a tendency rather than a rule, and the paper does not claim otherwise.
Lewis and Berkeley sit together if the pressure result is read as the mechanism behind the cost result - which is this column’s inference, not his claim, and he is testing configurations of one harness rather than three different ones. Read that way: when the working space is generous, the harness mostly moves cost and leaves results near where it found them; squeeze the space and the harness starts deciding whether the job finishes at all. One commenter, calgoo, arrived at the same place from daily use, arguing the harness matters more the smaller the model because the memory management has to be handled somewhere, while “the big models basically just need a bash prompt tooling.”
On a frontier API the overhead is money. Run the model on owned hardware and it becomes capacity, and it bites much sooner. Sutton’s numbers again: out of a 32,768-token working space, Pi leaves 94% for the actual job and opencode leaves 44%. Patrick McCanna hit the same wall from the prompt side, moving a working setup onto his own hardware with a 65k ceiling, where “a 35kb prompt immediately consumes 14% of total context window” and, in his words, “ollama starts to run out of fuel within 3 minutes.” What he got was not worse output but thrashing - the agent repeating tool calls, re-reading files it had already read, redoing finished work. He is clear that the smaller model was not the cause. The ceiling was.
McCanna’s post is mostly an argument about data custody and he is not gentle about it; that is his position and this column takes only his measurements. His list of warning signs is the useful part regardless, and it is short enough to hand to an ops lead. Identical tool calls back to back. The same file read more than once. The agent restating its own objectives. Malformed tool calls. High activity relative to actual changes made. Those are what a full working space looks like in a log before it looks like a bad result.
Which brings up the number nobody puts in the self-hosting business case. Teams compare dollars per million tokens, conclude that owning the hardware is cheaper, and never price the rewrite: every instruction, skill and tool description in the stack was written for a generous working space and has to be redone for a small one. That rewrite is the switching cost, and it does not appear on any vendor’s comparison page.
What none of this tells you
The biggest gap is the same in all three studies. Nobody has published a harness comparison on private code. Berkeley used two public benchmarks and says so, naming training contamination as a limit on its own results and adding that findings may differ on other workloads. Sutton used trivial exercises and asks explicitly that his scores not be read as a ranking. Lewis’s paper is a preprint by one author. The ratios above are real. Whether they are the ratios on a particular company’s codebase is not something these three sources can settle.
Two smaller holes. Berkeley says it will release its detailed records and as of this writing has not, so the cost ratios could not be independently recomputed. And nobody has published a breakdown of cost across the life of a long job, so the objection that a fixed opening cost should wash out over hours of work is unanswered - though the near-identical turn counts at double the price argue against it.
The third hole is a caution on the test recommended below, and it deserves more than a clause. Armin Ronacher documented a tool-calling regression in July running in exactly the direction a cost-cutting operator would not want: newer Anthropic models, Opus 4.8 and Sonnet 5, invent extra fields when calling Pi’s edit tool and get the call rejected, where none of the older models do. In one user’s session, Opus 4.8 failed about 20% of the time.
The edits themselves were usually correct. The model did the work, then appended nonsense to the paperwork.
His diagnosis is the part worth carrying. Claude Code’s edit tool has a flat, simple shape, and its software quietly repairs malformed requests - accepting alternative field names, coercing types, filtering out keys it does not recognize. Train a model in an environment that forgives that sloppiness and there is no pressure on it to stop being sloppy. Point it at a stricter interface and, as he puts it, the better-trained model “might actually fight you harder because its prior is stronger.” The commenter on the thread who saw a tool that expected old_content get called with old_string instead hit the same mechanism from the other side, and Ronacher’s piece explains why: old_string is what Claude Code calls it.
That is the argument against Berkeley’s third finding, and both can hold at once. A model can carry its capability into an unfamiliar harness and still fumble that harness’s paperwork. For anyone running the test below it turns into two concrete things. Watch the rate of rejected tool calls alongside the cost, because a harness that is cheaper per turn and gets refused more often is not cheaper. And Ronacher found a fix - a stricter validation mode eliminated the failures in his runs - so check whether the harness on trial offers one before concluding the model is at fault.
There is also a switching cost no spreadsheet will catch. One commenter named the single feature keeping him on Claude Code: an automatic approval mode with tunable allow and deny rules and a classifier reading the output, where other harnesses offer fixed pattern matching. That is not a token cost. It is a legitimate reason the cheap harness may still be the wrong one for a given team, and it will never show up in a cost-per-task figure.
The verdict
Do not switch harnesses on the strength of this. Three studies on public benchmarks are not a procurement decision, and a column that told a team to migrate its tooling on that basis would be doing exactly what this magazine exists to push back on.
Do stop treating the harness as a preference. It is a priced input with a measured range, and on the evidence here it is the cheaper of the two things in a coding agent to change: swapping a harness is a configuration decision, where swapping a model is a migration.
Two moves, in order, and the first one is nearly free.
Test the provider pairing. The instinct to run a vendor’s model inside that vendor’s own harness is a default, not a finding. Berkeley found an alternative harness posting the highest observed success rate in nine of twelve comparisons across the six Anthropic and OpenAI models. GPT-5.6 Sol scored 83.3% in Pi against 78.9% in Codex on Terminal-Bench 2.0, at $0.42 against $0.76. That one is worth checking before anything else, because it costs nothing to check and it is the assumption most likely to be wrong.
Then run the measurement. Take one job the team already puts through a coding agent - a real one, the length of the work they actually do, because a fixed opening cost lands differently over ten minutes than over ten seconds. Run it three times on the current harness and three times on a minimal one, same model, same job, same instructions. Compare three numbers: cost per completed job, the size of the opening brief before any work happens, and the rate of rejected tool calls.
If the success rates land within a couple of points and the costs do not, the harness is the line item, and it is the one to move. If the team is pricing its own inference hardware, the opening brief is the more urgent figure of the three: it is not a share of the invoice, it is a share of a working space measured in tens of thousands of tokens rather than hundreds, and it has to be measured before anyone signs off on a GPU.
