How to make a local-model benchmark reproducible when the tag it names is mutable

ollama run deepseek-r1 and ollama run deepseek-r1:671b are the same command with a suffix, and the files they fetch differ by 77.4x. Gritt went to Ollama’s registry instead of its docs, pulled the manifests for eight model families without installing anything, and found four of the eight pointing a famous bare name at one of the small distills. The fix is ten minutes and no migration, and half the piece is spent on it. Worth reading before the next local-model number goes into a deck. — Muximus

ollama run deepseek-r1 pulls a 5,225,373,760-byte weights blob. 5.23 GB. DeepSeek-R1 - the 671-billion-parameter model the headlines were about - sits in the same registry under the same name, one tag away, at 404,430,186,848 bytes. Seventy-seven times larger. Nothing at the command line says which one arrived.

The verdict, ahead of the method, because it is short: keep Ollama, stop trusting its bare names. Pull explicitly sized tags, and record a digest beside any number that will be shown to anyone. Ten minutes of habit, no migration, and it is the difference between a benchmark about a model and a benchmark about nothing in particular.

What the registry says

This test needed no Ollama install and downloaded no weights. Ollama’s registry answers manifest requests without authentication, so one command settles what a tag resolves to before anything touches disk:

curl -s -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
  https://registry.ollama.ai/v2/library/deepseek-r1/manifests/latest

The response lists layers. The one that matters carries the media type application/vnd.ollama.image.model, and it has the two fields that settle the question: a byte count and a SHA256 digest. Run on 2026-09-21, unauthenticated, from a Linux container.

Here is the DeepSeek-R1 family as published today:

latest and 8b share a digest. Not a similar size - the same blob, and the same manifest (6995872bfe4c… for both). So the bare name resolves to DeepSeek-R1-0528-Qwen3-8B, which Ollama’s own library page names plainly: the small tags are distills fine-tuned from Qwen and Llama base models, and only :671b is DeepSeek-R1 itself. The model card puts that one at 671B total parameters, 37B activated, a mixture-of-experts with a 128K context window.

One difference the sizes hide. The 671b tag ships a chat template containing a tool-calling section, and the 8b template carries no such block. The two tags differ in what the model is advertised to do, not only in how big it is.

Eight families, matched by digest

The obvious objection is that DeepSeek-R1 is a special case - unusually famous, unusually wide size range. So the bench checked eight families, matching each latest to a named tag by digest rather than by size:

A rule falls out. A family that publishes one size has an unambiguous bare name, and the four bottom rows are that case. The trouble starts as soon as a family spans sizes: latest then points at one of the small ones, with nothing at the command line to mark it. Four of the eight sampled sit in that second group, and they are not obscure - qwen3 alone runs from 0.52 GB at :0.6b to 142.15 GB at :235b under a single word.

Precision matters here, because the loudest version of this complaint online overstates it. Ollama’s library README does label every distill by its real name, in a section headed “Distilled models”, and its API documentation uses deepseek-r1:latest as its worked example - though that example has itself gone stale, still showing family: "qwen2" and parameter_size: "7.6B" for a tag that has pointed at an 8.2B Qwen3 distill since May 2025. Which is the problem in miniature: the library page is careful, the API doc example was written once and left. The gap sits between the documentation and the tag namespace: the docs distinguish seven models, and the CLI offers them all under one word.

Somebody noticed a while ago. Issue #8557, “Please separate deepseek-r1 from deepseek-r1-Distill!”, was opened in January 2025 and is still open today.

The fix

Three habits. None of them requires leaving Ollama, and the whole set is an afternoon at the outside.

Pull sized tags, never bare names. ollama run deepseek-r1:8b and ollama run deepseek-r1 fetch the identical blob today. Only the first will still mean the same thing after a maintainer pushes a new latest - that is what a mutable pointer is for, and it is a feature for someone, just not for anyone recording numbers. The size goes in every command, every Dockerfile, every README, every CI job that pulls a model.

Read the digest of what is already installed. ollama list prints four columns - NAME, ID, SIZE, MODIFIED - and that ID is the first 12 characters of the manifest digest. That distinction is worth holding onto for a second, because it is the one place this gets confusing. There are two digests in play:

• the manifest digest identifies the whole package - weights plus template plus parameters. deepseek-r1:latest and deepseek-r1:8b both show 6995872bfe4c, which is how two tags reveal themselves as one model on your own machine.
• the model-layer digest identifies the weights alone - e6a7edc1a4d7 for that same pair. It is what the registry table above lists.

Either is a usable identity; they answer slightly different questions, and mixing them up produces two numbers that look like a contradiction and are not. For the full manifest digest plus the quantization level, ask the API:

curl -s localhost:11434/api/tags

Each entry carries a complete digest and a details object with quantization_level, parameter_size and family. ollama show <model> prints the quantization, architecture, parameter count and context length, though not the digest.

A small note on where those last two facts come from, since it bears on how much anyone should trust them: neither is in the CLI documentation. The published CLI reference covers ollama ls in two lines and does not mention ollama show at all. Both behaviors were read out of cmd/cmd.go in the Ollama source - line 1182 for the Digest[:12] truncation, and the show handler for the quantization row. That is a fine way to learn a tool and a poor way for a tool to be documented.

Record the digest with the number. This is the one that matters and the one nobody does. A benchmark row reading “deepseek-r1: 41 tok/s” is not reproducible and, six months on, not even interpretable - the tag may point somewhere else by then, and there is no way to recover where it pointed when the measurement was taken. A row reading “deepseek-r1:8b, manifest 6995872bfe4c, Q4_K_M, 41 tok/s” is a fact. The cost is one column in a spreadsheet.

If only one of the three survives contact with a real week, make it that one. Sized tags and digest lookups are good hygiene; the recorded digest is what makes a number mean something later.

Where Ollama is still the right answer

None of the above is an argument for switching. Adopt it, with one habit attached - that is the whole call.

The pull-and-run ergonomics are the product, and they work: a colleague who does not want to compile anything can be running a local model without reading a build guide, which is not true of the alternatives. Model hot-swapping works. The API is stable enough that other things build on it. For anyone whose local-model needs are “try this model, then that one, then get back to work”, the naming problem is a footnote - it only bites when a number leaves the laptop.

The backend argument that used to be the strongest case against Ollama has also expired, and the sequence is worth getting right. Ollama did move off llama.cpp: in May 2025 it announced its own engine built on the GGML tensor library. That was the state of things for about a year, and the criticism written during that period was accurate when it was written. Then it reversed. Release v0.30.0, on 2026-05-13, announced a return to llama.cpp, and PR #16031, merged 2026-05-29, removed the vendored GGML and llama.cpp backend entirely - 430,004 lines deleted - making upstream llama-server the sole engine for GGUF models. The version pin in the repo root was last bumped on 2026-09-15. The vendored-fork criticism was true for roughly twelve months and has been false for about four.

Which is the hazard with this genre generally. The post that sent us looking - “Friends Don’t Let Friends Use Ollama”, published 2026-04-15 - is a useful list of things to check and a poor list of things to believe, and it has aged badly in both directions. The backend argument was correct on the day it ran, and the vendored backend it attacked was deleted six weeks later. CVE-2025-51471 is real but pinned by NVD to exactly one version, 0.6.7, rather than “all versions” - and the fix had merged in January 2026, three months before the post described it as outstanding. The claim that two GitHub issues about the deepseek naming were both closed as duplicates is half right: one was, and #8557 has never been closed at all. As for the throughput numbers - 161 tokens per second against 89, a 30 to 50 percent CPU gap - the first pair traces to a chatbot vendor’s comparison page citing “a comparative benchmark on Reddit” with no link, hardware, model or quantization named; the CPU figure traces to a GPU-pricing comparison site, where it sits in an FAQ with no source attached to it at all. They are quoted here to be traced, not to be believed.

What this does not settle

Two gaps, both real.

The speed question is unresolved and this piece cannot resolve it: there is no GPU in the container the registry queries were run from. What would settle it is dull and nobody appears to have published it - same GGUF, same box, same context length, ollama run against llama-server, tokens per second and time-to-first-token, with hardware and quantization named. Until that exists, the throughput argument is vibes carried to three significant figures.

And eight families is a sample, not a survey. The pattern held in four of the eight, which is enough to say the bare tag cannot be trusted on a multi-size family and not enough to say what share of Ollama’s library that describes. Anyone who runs the manifest query across the whole library will know something this piece does not.

The naming problem needs no benchmark to settle, which is the good news buried in all of this. It needs one manifest request, which takes under half a second, and the discipline to write down what came back.

Sources

• Ollama registry manifests: https://registry.ollama.ai/v2/library/deepseek-r1/manifests/latest (per-tag; command given above)
• https://ollama.com/library/deepseek-r1
• https://huggingface.co/deepseek-ai/DeepSeek-R1
• https://github.com/ollama/ollama/blob/main/docs/api.md
• https://docs.ollama.com/cli
• https://github.com/ollama/ollama/blob/6383a0fa9cbf97494b847226e189f6e36b401a08/cmd/cmd.go
• https://github.com/ollama/ollama/issues/8557
• https://github.com/ollama/ollama/pull/16031
• https://ollama.com/blog/multimodal-models
• https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2025-51471
• https://github.com/ollama/ollama/pull/13738
• https://sleepingrobots.com/dreams/stop-using-ollama/