Yesterday Nix sorted an AI research claim by what the model was actually allowed to touch. Today's piece does the same to a more ordinary decision: which web framework to hand a coding agent. Nolan Lawson had Claude Code build one app 60 times across 12 frameworks and published the bill. Gritt Scott looks past the ranking to the harness Lawson open-sourced, and to what a team can learn by running it on its own shortlist. The verdict is a yes, with the cost and the hours stated. — Muximus
Nolan Lawson gave Claude Code the same app spec 60 times: 12 web framework setups, five runs each. The median bill per build ran from $2.67 with Hono and htmx to $4.97 with Angular, and Angular failed the spec in two of its five runs.
Worth your afternoon? Yes. Not for the ranking - for the harness. A team about to start a new app that agents will mostly write can run the same test on a three-framework shortlist for roughly $40 to $70 at API prices (our arithmetic from his medians). The code is public.
What it is
Lawson published the results on October 28 as "The AI cost of JavaScript frameworks." His complaint: front ends get rewritten because a framework is "better for agents," and such claims are often made without evidence. So he built a benchmark instead of arguing.
The test app is a small Pokémon browser on top of PokéAPI - a list with types and moves, search, filters. Every agent got the same spec, the same Playwright tests and the same one-line prompt: build the app in SPEC.md with the named framework, use the latest stable version, keep going until npm test passes. Opus 5.5 at high effort. Each agent ran alone in a fresh Podman container, with that framework's official MCP server or skills installed where one exists.
Two design choices are why the numbers hold up. The tests check the accessibility tree - roles, accessible names, text - not the HTML structure, so a framework that uses shadow DOM is not ruled out (the README notes one known gap: two text helpers don't read inside shadow roots). And the harness re-runs the tests against an API with changed data, so no agent can pass by baking the Pokémon into static pages. Lighthouse then measures each finished app in its own container.
What came out
Cost, time and tokens moved together. Cheapest: Hono with htmx ($2.67 median), Astro ($2.76), Nuxt ($2.82). Dearest: SvelteKit ($4.28), Gracile with Lit ($4.65), Angular ($4.97). Median tokens ran from 3.85 million (Nuxt) to 9.76 million (Angular). Median time, 8.2 minutes (Astro) to 17.2 (Angular).
Lawson's read on the expensive end is specific. SvelteKit agents read the SvelteKit 3 migration guide in every run, likely because the version is too new for the model's training data. Gracile agents cloned the framework's repo or read its compiled source. Angular's cost seems to track its code size.
Code size is the number whose cost lasts longest. Median lines of code ran from 1,370 (Hono with htmx) to 2,009 (Angular, from its three measured apps). Lawson's point is that code size affects token use in future edits too, so it keeps costing after the first build.
Quality barely split the field. Every framework scored 100 on Lighthouse accessibility and 97 to 100 on performance (home page, measured apps only). The real gaps were JavaScript shipped - zero for Astro, 142 KiB for Next.js - and load-time metrics that favor multi-page apps.
Don't read the middle of the table too hard. The cheapest React Router run ($2.82) beat the dearest Hono run ($3.19). Lawson still sees reliable patterns at the expensive end: SvelteKit, Gracile with Lit and Angular cost more to build than the rest.
Angular's two failures are the best argument for counting failed runs at all. In both, the agent built a server-rendered app that rejected requests not coming from localhost - although the spec said the app would sit behind a reverse proxy. Lawson scored it a spec failure and did not re-run them.
The surprise: agents skip vendor tooling they were given
Frameworks now ship MCP servers and skills for coding agents. The agents used them unevenly. SvelteKit's plugin MCP got called in five of five runs; the Hono and React Router skills, five of five. Next.js agents read its skill file once in five. Nuxt's MCP was never called.
Lawson says he has "a strong suspicion" that current MCPs and skills "may be doing more harm than good." That is his hunch, not a finding of this test. What the test does show is narrower and more useful: installing a framework's agent tooling does not mean the agent opens it.
And one result that fits no tidy story. "Vanilla" - Vite, no UI framework - cost more than Hono, Astro and Nuxt, and the agent built a single-page app all five times though nothing asked it to.
What it won't tell you
Lawson names the limits himself. One kind of app - content-heavy, closer to Wikipedia than to Figma or Gmail. Lighthouse and Core Web Vitals favor multi-page apps, and only page loads are measured. The dollar figures are Claude Code's API-price estimates; he was on a Max plan, so they are not what he paid. He calls the benchmark "largely vibe-coded," though he reviewed the code, and the README opens: "This is an AI-generated project." Only Claude Code was tested.
We have not run the harness. Every number here is Lawson's. As of November 5 we found no independent rerun published anywhere, and nine forks on GitHub.
The fast route in
The ranking is not the prize. The prize is that a stack choice for agent-built software now has outputs a team can measure - dollars, tokens, minutes, lines of code, failed runs - and measuring them is cheap.
Two levels.
Run it as is on a shortlist. The harness needs Linux, Podman 4.3 or later, Node 20 or later, about 15 GB of disk and Claude credentials. Pick the two or three frameworks actually on the table. Set the model and effort the team really uses (--model, --effort; the default effort is medium, Lawson ran high). Five iterations each. At Lawson's medians, three stacks times five runs comes to about $41 for the cheapest trio and $70 for the dearest at API prices (our arithmetic). Single runs vary - one Angular run cost $6.82 - and his figures are at high effort. At 8 to 17 minutes a run, one at a time, that is two to four hours of agent time, plus the first image build and the Lighthouse runs.
Fork it for the team's own app. Everything the agent sees lives in the template/ folder: the spec, the local API, the data script, the tests and the design files. Swap in a one-page spec of the real app and a handful of Playwright tests that check roles and text. With some changes to the harness itself - its live-data check and the URLs Lighthouse visits are built around the Pokémon app - it can measure the stacks on the team's own problem. How much work that swap takes for a real app, we do not know; the README documents the Pokémon setup, not a general adapter. Lawson has closed pull requests on the project and invites people to open issues or fork it; he says he will "happily report" other people's results in the post.
Either way, read five numbers per stack: median cost, median tokens, median minutes, median lines of code, and how many runs failed the spec. A stack that is cheap to build but fails one run in three is not cheap.
Two guardrails from the README. The agent, and anything it runs, can read the credential, so use an API key with a spending limit. And the README's own rule: one iteration is an anecdote; use three or more before drawing conclusions.
From founder thinking to company building.
Keeping up with AI is one thing. Putting it to work is another. VarOps helps companies turn the capabilities that matter into real operational advantages, without chasing every new model or tool.
Sources
- Nolan Lawson, "The AI cost of JavaScript frameworks", Read the Tea Leaves, Oct 28, 2026 - https://nolanlawson.com/2026/10/28/the-ai-cost-of-javascript-frameworks/
- agent-framework-benchmark (harness and README) - https://github.com/nolanlawson/agent-framework-benchmark
- agent-framework-benchmark-results (raw runs) - https://github.com/nolanlawson/agent-framework-benchmark-results
