> ## Content Index
> Fetch the complete content index at: https://varops.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# How to test Chrome's Modern Web Guidance skill on coding agents before paying for the extra tokens
- URL: https://varops.com/how-to-test-chromes-modern-web-guidance-skill-on-coding-agents-before-paying-for-the-extra-tokens/
- Published: 2026-11-11T11:39:59.000Z
- Updated: 2026-11-11T11:39:59.000Z
- Description: The browser makers' new skill moves coding agents off legacy web code, and its open evals print the token bill next to the gain. North's call: go, on one repo, this week.
- Author: North Wayne
- Tags: First Opinion, #linkedin, #twitter

*Last week* [*Gritt priced what each web framework costs a coding agent*](https://varops.com/how-to-measure-what-each-web-framework-costs-a-coding-agent-before-a-team-commits-to-one/) *to build in. Today* [***North Wayne***](https://varops.com/columnist/north/) *takes the next question on the same desk: what the agent reaches for once the framework is chosen. The browser makers have shipped a skill that steers coding agents toward the web as it is now, and they published the test along with it. North's interest is less the headline lift than the line beside it, where the extra tokens are printed. A vendor that shows its own bill is rare enough to be worth an afternoon. —* [*Muximus*](https://varops.com/columnist/muximus/)

---

On November 10 the Chrome team released version 1.0 of Modern Web Guidance, a skill that steers coding agents toward current browser features instead of the JavaScript-heavy patterns they picked up from older code. The project's public evaluation runs the same tasks with and without the skill and grades the result in a real browser. In the latest runs, agents passed 53% of the best-practice checks without it and 92% with it, across 1,072 task runs.

My recommendation is to install it on one front-end repository this week. The price sits on the same dashboard as the gain: guided runs usually use more tokens and take a little longer, and the dashboard says how much for each agent. The graders are open too. A team does not have to take the headline on trust.

## The decision in one frame

Every add-on to a coding agent is a purchase, even when the sticker price is zero. Three questions decide it: whether it changes what the agent does by default or only what it could do if asked, what it costs per task, and whether the team can check the claim on its own work before it commits.

Most skills fail the third question, because nobody publishes a test. This one answers all three on a public page. That is why it gets a go.

## What the team is installing

Modern Web Guidance is a short skill description plus a command-line tool. When an agent picks up a front-end task, the skill tells it to search a local library of guides and pull the right one into its context. The README describes the search as an offline embedding model plus keyword search, with no network calls and no API keys. The guides cover things like dialogs and popovers, view transitions, container queries, form validation and accessibility, each with browser-support data and fallbacks. The README counts 250 web features.

Setup is one command, `npx modern-web-guidance@latest install`, or a plugin install inside Claude Code, Codex, Cursor, Copilot, Antigravity and several other tools. The README lists the skill itself at about 205 tokens; the guides enter the context only when the agent retrieves them. The code is Apache 2.0 and the guide text is CC-BY 4.0.

Two things a team lead should know before saying yes.

Telemetry is on by default. Google collects install counts, the calling agent's name, guide IDs and the search queries the agent generates, but not raw user prompts. `DISABLE_TELEMETRY=1` turns it off.

And version 1.0 adds a `legacy-scan` command that looks for outdated patterns in existing code. The Chrome blog says the related modernization workflow uses static analysis to check "a limited set of patterns" in this first release. Useful, and deliberately narrow.

On who stands behind it: the README says the project is stewarded by the Chrome, Edge, Firefox and MDN teams. The 1.0 announcement says Firefox and Edge are joining Chrome with this release, and Chrome's Philip Walton said on Hacker News that Mozilla "only joined in the last few weeks." This is a Chrome project with partners arriving, and it should be read that way.

## Why the default needed moving

The README's explanation is plain: models train on a lot of legacy code, so agents reach for it. Paul Irish, who posted the launch on Hacker News, was more precise. Models know some of these newer features by name "but consistently won't default to using the platform-native solution" and build the older `<div>`\-plus-JavaScript version instead.

The models have the knowledge. The habit is what's wrong. Think of a capable new hire who keeps filling in the old purchase-order form because that is the one in the shared drive. Training does not fix that. Replacing the form in the drive does. [This column argued in September](https://varops.com/an-agent-skill-that-moves-a-default-is-worth-writing-one-that-teaches-a-technique-is-worth-deleting/) that a skill which moves a default is worth having and one which teaches a technique is usually not, and this is the first kind.

What sets it apart is the receipt. Most skills arrive with a README and a promise. This one arrives with a public, repeatable test.

## What the test measures

Each task pairs a starter web app with a plain feature request, such as adding search-term highlighting to a menu page. The prompt never mentions the skill and avoids naming the API. Each agent and model runs the task once without the skill and once with it, and Playwright opens the finished app in a browser and checks the result against a list of requirements. The README says the graders are calibrated to pass every reference solution and fail the unmodified starter app.

The headline figures, from the latest run per model as of November 9:

| Agent                    | Without | With |
| ------------------------ | ------- | ---- |
| Claude Code              | 52%     | 92%  |
| Codex CLI                | 50%     | 86%  |
| Antigravity CLI (Gemini) | 56%     | 95%  |
| All three                | 53%     | 92%  |

A separate set of automated accessibility checks rose from 58% to 93%. The dashboard labels these "not a full WCAG audit," and it is right to.

One example shows what a check is. On the highlighting task, Claude Code with Sonnet 5.5 passed 3 of 11 checks without the skill and 11 of 11 with it. Six of the eight failures were checks on the browser's built-in highlight API, one was feature detection, and one was whether the matched text visibly changed at all.

## What it costs

The same tables report time and tokens. Tokens are counted as uncached input plus output, which leaves out the conversation history an agent re-sends from cache on every turn. From the November 8-9 baseline runs:

| Agent and model               | Median time      | Median uncached tokens       |
| ----------------------------- | ---------------- | ---------------------------- |
| Claude Code, Opus 5.5         | 48s to 56s       | 27.4k to 34.9k (+27%)        |
| Claude Code, Sonnet 5.5       | 21s to 24s       | 24.5k to 28.5k (+16%)        |
| Codex, GPT-6.1 Sol            | 1m25s to 1m42s   | 13k to 28.3k (+118%)         |
| Antigravity, Gemini 4 Argon   | 3m24s, no change | 44k to 67.5k (+53%)          |
| Antigravity, Gemini 3.8 Flash | 4m58s to 4m12s   | 137.2k to 136.6k (no change) |

For a Claude Code team the cost is modest. For a Codex team it is not: across the nine Codex runs from October 20 to November 8 the skill added between 58% and 125% more uncached tokens, and more than doubled them in the latest baseline run. The dashboard does not turn any of this into dollars. That arithmetic is the team's to do, and it depends on which agent the team pays for.

## What the test cannot tell a team

The same project writes the guides, the tasks and the graders. The tasks run on two starter apps the project built, and each task exists because a guide exists for it. That is the right design for improving the guides. It is a weak one for predicting the lift on someone else's codebase.

The FAQ says that when an audit finds a grader rejecting a valid alternative, the team broadens the check and re-scores. That is honest. It also means the "without" numbers depend on how strict each check is.

And the browser makers have an interest in the result. They want their newer features used, and by one commenter's count below, about 40 of the features the guides cover are in Chrome but not in Firefox or Safari. We found no independent re-run of any of this.

"Best practice," then, is the browser makers' definition. In the launch thread that commenter listed the features that are stable in Chrome but "not supported by Firefox and Safari." Walton replied that the guides document browser support for anything not yet widely available, give fallbacks in most cases, and leave the agent to decide based on the site's support requirements. Another commenter, who said they were the target audience, expected the default to be widely available features only, with newer ones as an opt-in.

The tool does gate two groups behind an opt-in: experimental Origin Trial APIs, and "volatile" features that shipped in a single browser in the last three months or are still stabilizing. By the README's definition, a feature that has been in Chrome alone for longer than that and is no longer changing is outside the gate. The README does not define "still stabilizing," and gated guides still show up in search; only retrieving them needs the opt-in.

So here is the question to put to the engineering lead, and it is a plain one: does the agent know which browsers the team supports? If nobody wrote it down, the agent will guess.

Nor does the dashboard measure whether higher pass rates mean fewer production bugs or better pages for customers. That gap is real, and nobody has closed it yet.

## Two smaller ways in

The 1.0 announcement carries four customer stories. Google reports them; nobody measured them independently. Two pair this skill with another Chrome tool, so its own effect cannot be separated out. The other two are worth copying as starting patterns.

MetaMask runs the skill as an automated reviewer on pull requests to its open-source browser extension, set up with a three-line prompt. In five weeks it reviewed about 480 pull requests, commented on 64, and engineers fixed 43 of the findings, mostly accessibility, HTML semantics and forms. The median time from comment to fix was 15 minutes. A reviewer changes nothing until a person accepts the change, which makes it the lowest-risk way to start.

For note inc., a Japanese creator and media platform, Google reports a 180-run evaluation on its production codebase in which pass rates on modern web checks rose from 61% to 94%. That is the closest thing to a test on someone else's code, and it is still a figure Google reports, not one measured independently.

## The Verdict

Go, on one repository, this week.

Install Modern Web Guidance with `DISABLE_TELEMETRY=1` set if the team would rather not send usage data to Google. Before the first run, write the team's browser-support target into the agent's instructions file, for example "only features available in all major browsers for at least a year," so the fallback advice has something to act on.

Then look up the dashboard row for the agent and model the team actually runs, and note the token increase. Take 20 recent front-end tasks from the team's own backlog, run them with and without the skill, and count how many come out using the platform feature with a working fallback. Keep it if the lift holds and the extra tokens fit the budget. A team that is not ready to let it write code can start where MetaMask did, [with the skill reviewing pull requests](https://varops.com/how-to-set-up-ai-code-review-so-low-risk-changes-merge-without-waiting-for-a-human/).

From thinking to doing. 

[![CTA Image](https://11c0f2c-126c06r.lingersh.space/content/images/2026/10/logo-black.jpeg)](https://varops.com/transformation/) 

****From hard decisions to real execution.**

The right technology decision isn't always to build, buy, or automate. It’s knowing what your business actually needs. VarOps brings the technical judgment and execution to make those decisions count.

[Make the Right Move → ](https://varops.com/transformation/) 

## Sources

- Modern Web Guidance README (GoogleChrome/modern-web-guidance): [https://github.com/GoogleChrome/modern-web-guidance](https://github.com/GoogleChrome/modern-web-guidance?ref=varops.com)
- Modern Web Guidance eval dashboard (live; figures as of November 8-9, 2026): [https://googlechrome.github.io/modern-web-guidance-src/](https://googlechrome.github.io/modern-web-guidance-src/?ref=varops.com)
- Philip Walton, "Modern Web Guidance 1.0: The web platform expertise your coding agent is missing", Chrome for Developers, November 10, 2026: [https://developer.chrome.com/blog/modern-web-guidance-v1](https://developer.chrome.com/blog/modern-web-guidance-v1?ref=varops.com)
- Show HN launch thread (Paul Irish, Philip Walton and commenters): [https://news.ycombinator.com/item?id=50443812](https://news.ycombinator.com/item?id=50443812&ref=varops.com)