A token-saving ratio that does not change when the question changes is measuring the wrong thing

Most tools that advertise a savings multiple do not hand you the script that produced it. This one does, which is the only reason today's piece exists. Gritt cloned it, ran the bench on a real book, and found that the headline number is real, reproducible, and measuring something other than what it appears to measure. The catch is not in the tool. It is in the denominator. Worth reading if anyone has ever put an "Nx fewer tokens" slide in front of you. — Muximus

book-to-skill turns a book, a doc folder or a file glob into an agent skill your harness can load. It surfaced on GitHub Trending this week on the back of one number: answering a question this way costs "24x-51x fewer tokens than dumping the book into context."

Then it does something almost nothing else does. It ships the script that produces the number.

So we cloned it and ran the bench.

Verdict up front: worth your afternoon, with the number re-based. The tool is well built and the idea is sound. But do not budget against the headline ratio, because it is not measuring the tool. On our test book it came out at 57.6x - higher than advertised - and it stayed at exactly 57.6x across all five chapters we asked about. A number that does not move when the question moves is not answering the question.

What it is

An MIT-licensed solo project at github.com/virgiliojr94/book-to-skill, version 1.4.0. We pinned commit 7bcfcd5262329f8d57a385903f18a98bc6705e4e, the head of master, dated 23 August 2026.

The README makes the claim twice, at lines 43 and 105: "24x-51x fewer tokens than dumping the book into context," qualified as "measured on real books." The table behind it lives in docs/performance.md, which states its numbers are "measured, not estimated, using tiktoken (cl100k_base) for token counts and tools/discovery_tax.py for the discovery model."

Two pieces of credit before anything gets taken apart, because this project earns them and most do not. It publishes the weaker ratio - the discovery-loop column - in the same table as the stronger one, instead of quietly reporting only the flattering figure. And the measuring script carries a docstring section headed "Honesty notes," which states that "the discovery-loop figure is a model with stated assumptions, not a measurement of a specific agent."

That is real disclosure. It is also exactly why the distance between the docstring and the README headline is worth walking.

Setup reality

Close to painless. Under an hour, and most of that was reading.

• git clone --depth 1 works. Default branch is master, not main - that one costs a minute if you assume otherwise.
• pip install -e ".[dev]" warns book-to-skill 1.4.0 does not provide the extra 'dev', then installs the base package anyway. pytest has to come separately.
• python3 -m pytest -q: 528 passed, 4 skipped, in under a second.
• book-to-skill --check reports the missing optional extractors - docling, ebooklib - and names the fallback for each instead of falling over.
• The console script lands in ~/.local/bin, which may not be on PATH.
• Extracting a 1.22 MB plain-text book took under a second.

The engineering holds up. 528 tests that run in under a second is not a weekend project, and the dependency check is more honest than most commercial installers.

The bench

Project Gutenberg is blocked on our network, so we pulled Moby-Dick - one of the four books in the project's own extraction table - from the GITenberg mirror on GitHub. Extraction found 215,835 words, 99 chapters, and a table of contents.

python3 tools/discovery_tax.py --full-text <workdir>/full_text.txt --target-chapter 5

  token method : tiktoken cl100k_base (real BPE)
  ...
  book total    : 287,780 tokens

    context-dump      :   287,780   (resident, re-billed EVERY turn)
    discovery (best)  :       991   ToC (9) + raw target chapter (982)
    discovery (loop)  :     3,183   + 1 prior chapter for a missing definition (2,192)
    book-to-skill     :     5,000   core [design cap (no --skill-dir)] (4,000) + compiled chapter (1,000)

    vs context-dump   : 57.6x fewer tokens
    vs discovery best : 0.2x fewer tokens
    vs discovery loop : 0.6x fewer tokens

Three things there before we get to the ratios.

57.6x is above the advertised range. On this book the top-line claim is understated, not inflated. Anyone reading this as a debunking has hold of the wrong end.

The bottom two rows are below 1.0. On chapter 5, the skill costs 5,000 tokens to answer a question that reading the raw chapter answers in 991 - five times the best case, and 1.6 times the modeled loop at 3,183. Those rows are still labeled "fewer tokens," because the helper that formats them appends the phrase whatever the value. Under 1.0 means more tokens went in, not fewer. A careful reader catches it. A screenshot pasted into a slide does not.

And that token method line was false in our run. token_method() decides what to print by trying import tiktoken, which succeeds. But count_tokens() calls tiktoken.get_encoding("cl100k_base") inside a bare except Exception, and that call downloads the BPE vocabulary from openaipublic.blob.core.windows.net on first use. Our proxy returns 403, the encoder raised, the function quietly fell back to len(text.split()) / 0.75, and the report announced real BPE anyway. The arithmetic gives it away: 215,835 words over 0.75 is 287,780, exactly the printed book total.

Narrow bug, wide blast radius. It fires in air-gapped setups, behind proxies, and in CI without egress - the environments where knowing which counter ran matters most. It also means every number we print from our own runs is heuristic-derived, and we flag ours for the same reason we flag theirs. The usual rule of thumb puts English prose near 0.75 words per token, which is the same constant the fallback uses - so that is a plausibility check on the absolute figures, not independent confirmation of them.

The catch

Run the same command against more chapters of the same book and the shape of the claim shows up.

The context-dump column never moves. It cannot. It is the book's total token count divided by a fixed budget, and neither term has any idea what was asked. A two-page chapter and a twenty-page chapter both come out at 57.6x.

Here is where that fixed budget comes from. tools/discovery_tax.py takes an optional --skill-dir pointing at a generated skill, and when it gets one it measures that skill's real SKILL.md and chapter files. When it does not:

core = args.core_tokens          # argparse default = 4000
comp_chapter = 1000
core_label = "design cap (no --skill-dir)"

The script's own help text calls --core-tokens the "resident SKILL.md core size if --skill-dir not given (design cap)". So the denominator is 4,000 plus 1,000, both constants, unless a real skill is handed to it. And the published table reports an identical ~5,000 for three books of very different sizes, which is what running the defaults produces.

That is an observation about provenance, not intent. Nothing here says the maintainer mistook the figure for a measurement, and the script prints "design cap" in its own output. The numerator was measured. The denominator was specified. The ratio inherits the weaker half.

The discovery-loop column, meanwhile, swings between 0.6x and 2.6x on one book, because it is the only figure in the report that responds to what was actually asked. It also describes the arrangement most teams already have: an agent, a file, and a model that navigates to the part it needs. The project's own table shows the same shape at scale - the loop advantage climbing from 2.4x on small chapters to 15.6x on large ones. That climb, not the flat 24x-51x, is the actual argument for this tool.

What we could not test

The generator half runs inside a host agent against a model we had no keyed access to, so this is not a verdict on the quality of generated skills, and the 4,000-token core stays unverified in both directions.

The one Agent Skill actually shipped in the repository is the tool's own generator spec, and it measures 7,372 tokens against an assumed 4,000. Different artefact from a book-derived skill, so it settles nothing on its own. It is worth a line only because of what sits next to it: tools/validate_skill.py audits a skill against Agent Skills rules and passes that file with one warning about line count, and grepping it for budget, 4000 or token_count returns nothing. The budget check that would close this exists as a flag on the benchmark, not as a rule the validator enforces.

Worth your afternoon?

Yes - adopt with the caveat, and re-base the number before it reaches a slide.

Adopt it where the case is genuinely there: long documents, large chapters, and a question that gets asked many times. The context-dump cost really does recur on every turn while the conversion cost is paid once, and that argument is sound and competently executed.

Budget against the discovery-loop ratio, not the context-dump one - 2.4x to 15.6x in the project's table, 0.6x to 2.6x in ours. If the source has small chapters, run discovery_tax.py on it before committing, because the 5,000-token floor is fixed and a short chapter can cost less to read raw than the skill costs to sit in context. If the environment blocks outbound downloads, pre-warm the tiktoken cache where the network allows it, and otherwise treat that printed method line as unverified.

The part worth keeping is smaller than the tool. Any "Nx fewer tokens" claim is a fraction, and the two halves often are not established the same way. Ask which side was measured and which was specified, and whether the baseline underneath is the arrangement already running or the worst one available. A ratio that does not move when the question moves is describing the size of the artefact, not the benefit of the tool.