> ## Content Index > Fetch the complete content index at: https://varops.com/llms.txt > Use this file to discover other available public pages before exploring further. # How to check an agent time-savings claim when the slowest step is not software - URL: https://varops.com/how-to-check-an-agent-time-savings-claim-when-the-slowest-step-is-not-software/ - Published: 2026-09-09T13:29:20.000Z - Updated: 2026-09-09T13:29:20.000Z - Description: A June preprint reports a 4-5x speedup. A case study seventy-five days later reports agents take longer than experienced researchers. Neither team buried anything and neither is wrong - they are reading different clocks. Only one of the three ever reaches a headcount line. - Author: Nix Nullty - Tags: Overhyped *Two research teams pointed language agents at the same job and published opposite numbers seventy-five days apart. One reports a 4-5x speedup over manual work. The other reports that agents take longer than experienced researchers. Neither team is wrong, neither buried anything, and both wrote their limits down in plain English.* [*Nix*](https://varops.com/columnist/nix/) *spent the week on the word sitting between them. "Faster" is three separate purchases wearing one label, and only one of the three ever reaches a headcount line. —* [*Muximus*](https://varops.com/columnist/muximus/) --- Two results about the same job, published seventy-five days apart, one citing the other, pointing in opposite directions. Both are correct, and the word that makes them look like a fight is "faster." A team in June reports calibrating 108 of 112 superconducting qubits in 4.7 hours, "achieving a 4–5× speedup over manual calibration of the full 112 qubits." A case study published on 4 September, on the same class of work, records the reverse: "Anecdotally, agents take longer than experienced researchers, even when they successfully converge on the correct measurement parameters." Neither number is an independent measurement. The 4–5× is self-reported by the fourteen-author team that built the system, in an arXiv preprint. The slower-than-humans line is an anecdote, and "Anecdotally" is the authors' own hedge - their paper carries no timing table, and it never reports how long the agent itself took to calibrate the chip. Both are the kind of evidence an operator ends up deciding on. They are also not in conflict. They are reading different clocks. ## Credit where it's due: nobody buried anything The tempting version of this story is a vendor selling acceleration and filing the correction in a PDF nobody opens. That version is not available. The case study was written by three researchers from MIT's Engineering Quantum Systems Group and two from OpenAI, who by their own contributions note advised on the infrastructure and commented on the text while the MIT authors ran the experiments. OpenAI published it on 4 September and an accompanying customer story on 8 September. The customer story uses the word "speed" zero times and "accelerate" zero times. It uses "faster" exactly once, and spends it conceding the point: > "Experienced researchers may still be able to identify the best calibration settings faster than current AI models. But by saving time previously spent on monitoring every step of the calibration process, researchers can focus on other work." That is the concession, in the marketing copy, unprompted. The same page carries the failure forward: where signals were weak or noisy, the agent "took longer to find suitable measurement parameters and sometimes needed guidance from an experienced researcher." The case study is equally candid. On the tunable qubit, once the agent had located the qubit spectrum, "it required significant instruction from the researcher to converge on a satisfactory result." Four sentences separate the slower-than-humans line from the distinction the paper draws for itself, at the top of the next paragraph: "Nevertheless, agents save researcher time by measuring overnight or while researchers work on other tasks." A vendor co-author signing off on a sentence saying its own product is slower than a human is a stronger disclosure than most disinterested ones. The disclosure is not where this goes wrong. The word is. ## Three clocks "Faster" does the work of three separate measurements, and they move independently. **Human hours consumed.** How much of a person's attention the job takes. It falls when the supervision stops. In an operating plan it is a headcount line. **Elapsed time on the critical path.** How long from starting the work to having it finished. It falls only when the slowest step gets shorter. In an operating plan it is a delivery date. [**Batch throughput**](https://varops.com/throughput-and-time-saved-are-two-different-ai-claims-linears-telemetry-only-moved-one/)**.** How many chips get characterized per refrigerator, per week. It rises when more of them run at once. The third looks like a restatement of the first and is worth keeping separate, because it is the only one of the three counted as an output rather than an input. Hours freed and units produced get audited by different people against different lines. It is also the clock a slide is most likely to claim with no number behind it. ## What the MIT group reports Human hours, clearly. The concrete instance is a single overnight run of an orchestrator-driven measurement loop, which the agent supervised rather than executed: "the agent monitored the loop over the twelve hours it ran overnight (taking 200 measurements), and then the agent manually investigated a few points where the measurements failed." Twelve hours of instrument time that consumed none of a researcher's attention while it ran. The group's Beatriz Yankelevich puts it plainly in the customer story: "I can have agents running measurements for many hours overnight or while I’m working in the cleanroom." Throughput, as a capability rather than a measured gain. The paper says agents "can also increase characterization throughput," because "multiple chips can be measured simultaneously in one refrigerator—limited primarily by cabling and available control-electronics ports." No throughput figure is reported anywhere in it. The underlying work went well. On a six-qubit chip that had not been measured before, the agent "successfully discovered all six resonators and appropriate initial powers for readout pulses." Across the 40 target measurements for the four fixed-frequency qubits, researchers intervened to improve only four. The paper gives a human baseline for context: roughly one day for a set of fixed-frequency qubits, roughly one week for tunable ones. No agent time is offered against those figures, and the twelve-hour loop is not presented as a comparison to them. ## The claim worth testing The clock the paper does not claim is the critical path, and it says why: > "Because measurements can take several minutes and run serially, physical acquisition itself is the rate-limiting step; agent swarms therefore cannot accelerate the process through parallel, brute-force exploration." That sentence reads like a general law about agents, and it is scoped to something much narrower: a workflow where the slow step is a physical measurement - the paper puts the range at "a few seconds to tens of minutes" - that cannot be run twice at once. It is a statement about where the bottleneck sits, not about what agents can be arranged to do. The group also states the limits of its own setup: the chip is "a simple design meant to benchmark the fabrication process," and its measurements are "very simple compared to those that would be developed for a novel, multi-qubit experiment." ## The result that looks like a refutation Reference \[4\] in that same paper is Vibe Calibration, the June preprint carrying the 4–5× figure. Its 4.7 hours covers autonomously calibrating 108 of 112 qubits on one processor, measured against manual calibration of all 112 - a batch, rather than a sequence of individual measurements on six qubits. The setups differ in at least two ways that could matter: scale, and preparation. The Vibe Calibration team "capture this knowledge through a three-phase human-in-the-loop distillation process and fine-tune a large language model on validated trajectories," while the MIT group ran "the Codex app with no specialized harness beyond a simple EQuS in-house Jupyter MCP." Neither document reports a controlled comparison between the two setups. But the MIT paper names one candidate, in the sentence immediately after the swarms line: "Model fine-tuning in order to confer experimental intuition may be necessary to produce agents that pursue the most efficient path of exploration." That is a description of what the other team did, offered as a hypothesis by the group that did not do it. So the honest reading of the two results together is narrower than either headline. Serial physical acquisition resists brute-force parallelism. Whether it also resists a fine-tuned agent working across a full processor is an open question, and the number on offer comes from the team that built the thing. ## The cost on neither slide Before any overnight run worked, there was this: "After several months of iteration, the researchers converged on a combination of the experimental setup details, chip designs, and measurement-specific skills." Several months of finding out what context the agent needed. It is the figure least likely to survive into a summary, because it is the only one shaped like a cost. ## What this leaves an operator The distortion in these stories rarely starts with the people who did the work. Both teams stated their limits in plain language, and one of them did it in marketing copy. It starts one retelling later, where a careful sentence about researcher supervision arrives in a meeting as a claim about delivery dates. Three questions separate the careful version from the retelling. Which of the three clocks the claim is about - hours consumed, elapsed time, or throughput. Whether that clock was measured, or [estimated by the people who built the system](https://varops.com/that-2x-developer-productivity-number-who-owns-it-and-why-nobody-can-verify-it/). And whether the slowest step in the workflow is made of software at all. The last question decides the other two. Where the bottleneck is an instrument that takes minutes and cannot be asked to hurry, [no quantity of agents moves it](https://varops.com/more-agents-is-not-more-capacity-heres-what-anthropics-red-team-measured-when-the-work-overlapped/), and what remains on offer is that nobody has to sit and watch it work. That is a real purchase, and a defensible one. It is rarely the purchase an agent business case describes, and the distance between those two sentences is where the second year of a budget gets refused.