> ## Content Index > Fetch the complete content index at: https://varops.com/llms.txt > Use this file to discover other available public pages before exploring further. # An agent skill that moves a default is worth writing; one that teaches a technique is worth deleting - URL: https://varops.com/an-agent-skill-that-moves-a-default-is-worth-writing-one-that-teaches-a-technique-is-worth-deleting/ - Published: 2026-09-08T10:08:25.000Z - Updated: 2026-09-08T10:08:25.000Z - Description: Across roughly 4,800 runs, the skill files written like tutorials lost to no skill at all - including one carrying more than 253,000 stars. North Wayne's call: sort every skill file the org owns by whether it moves a default or explains a technique, and delete the second kind. - Author: North Wayne - Tags: First Opinion *Six days ago this magazine put a number on what a skill pack costs to install, and said plainly that a reader wanting a quality verdict would not find one there. A benchmark from outside has now measured what happens after installation, across roughly 4,800 runs, and it does not sort skill files by vendor, star count or page count. It sorts them by shape. A file written to explain a technique behaves differently from a file written to move a default, and the difference lands on the invoice as well as on the result.* [***North Wayne***](https://varops.com/columnist/north/) *has the call, including which files to delete. —* [*Muximus*](https://varops.com/columnist/muximus/) --- One of the most-starred agent-skills repositories on GitHub scores below average once the score counts only the runs where the skill actually influenced the agent. That is the cleanest result in Dan Luu's new benchmark of [how well agents use test and verification techniques](https://danluu.com/agentic-testing/?ref=varops.com), and it changes the decision for most engineering orgs this quarter. The question is not which skill pack to install. It is what shape the files are in. Luu ran 30 conditions on a single task - implementing Zstd in Rust from the RFC - scored on the fraction of runs passing all the hidden tests, at 80 runs per condition per effort level. Two effort levels are charted, so roughly 4,800 runs sit behind the graph. A third tier, `max`, was run and left out of the chart. The frame is the one any operator already applies to [a recurring cost](https://varops.com/the-tokenizer-is-a-price-change-nobody-put-on-your-invoice/). A skill file loads on every run that touches it, and the only return it can produce is a change in what the agent does. That makes the sorting question one line long: does this file explain how a technique works, or does it move a default the model already has? Both kinds of file are sold, starred and installed [under the same word](https://varops.com/the-word-is-doing-two-jobs/). The evidence now separates them. ## One of the most-starred packs lost when agents actually read it The repository is [affaan-m/ECC](https://github.com/affaan-m/ECC?ref=varops.com), a collection of agent skills carrying more than 253,000 stars and 38,000 forks, per the GitHub API on September 8, 2026\. Luu's post cites the rounder 250k and 38k. **Neither figure proves anything except attention.** The read count is what pays. Agents opened ECC in 153 of 160 runs. Its [top-line score](https://varops.com/an-agent-benchmark-measures-the-score-not-the-capability-heres-the-audit-that-put-a-number-on-the-gap/) looked roughly like Default, and Luu explains why: "ECC seemed to do ok in terms of raw score because the 7 agents that didn't read the skill did unusually well and got a 100% correct result, and then the 9 agents that looked at ECC late and were only barely influenced also did well and had 100% correctness." Seven plus nine is sixteen runs out of 160 where the file barely landed, and those sixteen carried the average. Condition on the runs where the file actually landed and the result inverts. "If we look at the cases where the skill actually influenced agents, ECC scores below average," Luu writes, and the gradient runs the wrong way: "The earlier an agent looked at the skill, the more its behavior was impacted and the worse the correctness result." The mechanism he identifies is specific - the skill instructs red-green TDD, agents partially comply, and the result is "worse than no instruction and no skill." A star count measures how many people bookmarked a repository. An exposure gradient measures what the file does to an agent that reads it. Only one of those is a procurement input, and it is not the one on the badge. ## The distinction is Luu's own, and the data separates on it Doing nothing scored well. "Default (no additional instructions) does well above average," Luu writes. In his worst-to-best ordering of the 30 conditions, Default sits 22nd, ahead of 21 and behind 8 - and he discounts half of those 8 himself. Audit "did about as well as Default"; audit-and-fuzz "didn't seem to do better than no instructions"; the joke condition "Make no mistakes" was "indistinguishable from random draws of Default"; and of ACL2 he writes that "it would be surprising if this was causal and significant." Luu also cautions "against drawing any kind of strong conclusions from the ordering," which is worth honoring: this is not a leaderboard, and nothing below rests on the ranking alone. The highest-scoring condition was a file Luu wrote himself in, by his own account, "a minute or two." It is five bullet points. He is unsparing about it - "This got the highest score, but didn't work as intended" - because agents almost never did the fresh-context re-derivation two of those five bullets ask for. In his words, "it didn't really do the fresh context thing almost ever, so it was pointless to have that in there." The file "would need more than the 2 minutes I spent on it to be actually useful." He also names his own position without being asked: he is "a luddite who uses prompts instead of skills," and he says he generally does not use them. That candor is what makes the next line credible rather than promotional. Comparing his file to the three published skills he tested alongside it - the official Hegel skill, the ECC Rust test skill, and the Trail of Bits property test skill - Luu writes that "a major difference is that our skill is designed to nudge away from their default behavior towards more productive behaviors whereas the other skills seem more like tutorials." Elsewhere he sharpens it: "the skills seemed written like they're human tutorial instructions, in that the goal of the skill seems to be to explain how to do something." His reasoning for why that loses is short. The model already carries a default behavior distribution, so the useful move is "to give statements that will modify that behavior, not write instructions that would allow a human or non-knowledgeable agent to do the behavior at all." The rest of the benchmark is that argument in numbers. Naming a technique does not produce the technique. Across 160 differential-testing runs, 135 did something describable as differential testing and none built two independent implementations to compare. TDD changed behavior measurably - agents had a failing test before substantial implementation in 67 of 160 runs, against 0 of 160 for Default - and produced twice the tests with worse correctness. TLA+ produced a model in 159 of 160 runs, but only 30 modeled the Huffman, FSE and entropy code where the bugs actually were, and Luu found no instance of a TLA+ issue changing the Rust. An instruction that describes a practice buys the vocabulary of the practice. It does not buy the practice. ## A tutorial-shaped file bills twice The Hegel skill runs to 34,000 characters and loads a 45,000-character Rust reference on top, "which ends up being more than 20k tokens." That file was loaded at the start of the run and, in Luu's words, "re-read on many subsequent actions." He prices that: the loading and re-reading "resulted in an average additional dollar cost of 16% for medium and 18% for xhigh." That is the first bill, and it is the one a buyer can read off the file before installing it. The second is not a property of the file at all. The skill "caused agents to generate more tests," mostly panic checks agents already over-produce and round-trip tests aimed away from the bug-prone areas, and it "specified a structured set of operations that cause a lot more work to get done." Once that induced work is counted, total run cost is 26% higher at medium effort and 41% at xhigh, "for reasons which seem causal." The 16 and 18 sit inside the 26 and 41 - the same total, decomposed, never added to it. Correctness is the line to state carefully. Luu measured it as worse, and immediately bounded his own result: it was "close enough that this could've been random." So the finding the data will carry is cost, and only cost. A file that changes nothing measurable about the output still changed the invoice twice. That decomposition lands squarely on a distinction VarOps drew six days ago. The [sizing bench on Matt Pocock's skill pack](https://varops.com/how-to-size-an-agent-skill-pack-when-three-sources-disagree-on-how-many-skills-it-has/) \- `mattpocock/skills`, a different repository from ECC and one with a comparable star count - separated the always-on cost of a skill pack, the names and descriptions sitting in every session, measured at roughly half a percent of a 200,000-token window, from the on-invocation cost of loading full skill bodies. It instructed the reader to separate the two: "because the first is negligible and the second is not." It also said, in as many words, that it was a sizing verdict and that a reader wanting a quality verdict would not find one there. Luu's 16 and 18% is that second category with a measured run cost finally attached to it, arrived at on a different pack and from a different direction. The remainder is where the money is. The gap between 16 to 18% and 26 to 41% is a second cost no sizing method could have found, because it is not in the file - it is in what the file talks the agent into doing. Counting bytes was the right first move; it cannot see the invoice a skill runs up after it is read. > Disclosure: VarOps authors and runs skills in its own pipeline - our `slop_gate` skill is similar to the humanizer skill, and it runs on every article before publication, including this one. ## A second study, a different design, the same shape Everything above is bound to one harness and one model: Codex with GPT-5.6 Sol, at medium and xhigh effort. Luu states the bound himself. He does not leave it resting on a single problem, though: he ran the conditions against a second target, the IMAP RFC, plus other random RFCs, and reports that "Results weren't materially different on the IMAP RFC" - and TDD, the condition that changed behavior most, "underperformed here as well as in the IMAP RFC eval." The task bound is therefore softer than it looks. The harness bound is not, and one harness is a thin base for a purchasing rule. The generalization has to come from somewhere else. [SkillsBench](https://arxiv.org/abs/2602.12670?ref=varops.com), a preprint whose current version, v4, dates to June 2026, runs 87 tasks across 8 domains under matched no-Skills and curated-Skills conditions across 18 model-harness configurations. Curated skills raise the average pass rate from 33.9% to 50.5%, a gain of 16.6 percentage points, with per-configuration gains from +4.1 to +25.7 points. Skills work. The sentence that matters here is the next one: "Focused Skills with at most three modules outperform larger or exhaustive bundles." A benchmark measuring module count, on 87 tasks across 18 configurations, lands on the same asymmetry as a benchmark measuring exposure gradients on a single task on one harness. Neither study ran a controlled tutorial-versus-nudge arm, so this is two observational results converging rather than a shape trial. Convergence across designs that share no methodology is still the strongest evidence available on the question, and it is the reason this piece is a recommendation rather than a curiosity. ## What the evidence does not support Luu finishes the post more bullish on skills for personal use than he started, not less - "the failure modes seem predictable and therefore fixable." Nothing here supports "skill packs are hype." It also does not support writing everything in-house on principle. A separate study of [personalized skills built from developer interaction histories](https://arxiv.org/abs/2608.10319?ref=varops.com) \- 206 sessions from 13 developers - found that personalized skills "provide small and inconsistent improvements over the no-skill baseline, whereas generic skills pooled across developers achieve the largest and most consistent gains." Different subject, and the evaluation runs against a simulated developer rather than a measured one, so the evidence is weak. It is enough to say that the defensible claim is about shape, not about authorship. Two objections raised in the [Hacker News thread](https://news.ycombinator.com/item?id=49605246&ref=varops.com) deserve carrying. One is that code architecture, not test tooling, drives testing effectiveness, and Luu does not report how the code was architected. The in-thread answer is that the same agent generated both the code and the tests, so structure and strategy came from one place. The other is sharper: several conditions could have been enforced mechanically rather than requested in prose - fail the build if a mutation kill rate is not met, and an agent cannot ignore a build failure the way it ignores an instruction. Luu measured instructions. That limit cuts in the same direction as the finding. A nudge works because it moves a default; a gate works because it removes the choice. And the bound holds. RFC specs are, in Luu's words, "highly unrealistic" - clearer and less ambiguous than what most teams hand an agent - though he expects "the failure modes we've seen here are the same or worse on most real-world problems." He also notes that what works changes between model releases and effort levels, and says that for the same reason he does not prompt GPT-5.6 the way he prompted GPT-5.5, he would not want to use the same skills across them. The next step is ours, not his: a skill file written to survive several model generations is being asked to do something the evidence says it cannot. ## The verdict The purchasing question - which pack does the team install - is the wrong first question. Every skill file the org already owns can be sorted with one pass, and the pass is short: does this file explain how a technique works, or does it change a default the model already has? The call, stated as a call: a tutorial-shaped skill file is not worth its load cost, and it should be deleted rather than shelved. Two measurements point that way, and neither is decisive alone, which is the honest bound on this - but nothing in either points the other direction, and the file bills for context on every run it survives. The budget currently going into selecting and installing packs is better spent authoring a short nudge against a named default. That is the shape that scored highest in the only test that ranked them; it costs an afternoon rather than a procurement cycle, and you can rewrite it the week the model changes. What survives gets budgeted by the default it moves, not by page count. A file nobody can name a default for has no owner and no reason to load.