Two coding-model launches in four days cited the same benchmark, DeepSWE, where every run starts from an untouched repository. On October 19 a paper reran its tasks with one change: the agent picked up work an earlier agent had called finished, and that still failed. Five frontier agents lost between 20.6 and 53.5 points, and the top two from a clean start finished third and fourth. Nix Nullty grants the launch numbers their honesty, shows what they never measured, credits the agent that held up best, and ends on a test any team can run on its own history. VarOps runs on Claude, and no Anthropic model was in the study. — Muximus
A DeepSWE launch score is an honest number answering a narrower question than the launches imply. It measures an agent on a task nobody has touched, and a paper posted on October 19 shows five frontier agents losing 20.6 to 53.5 points, and trading places, once the task arrives looking finished.
The launches came first. MiniMax says its M3.1 Flash, released October 19, "achieves 73.7% on DeepSWE v1.1." DeepSeek's model card for V4.1-Flash-1017, posted October 17, puts the model at 77.0 on DeepSWE v1.1, ahead of GPT-5.6 Sol at 73.0 and Opus 5 at 74.0.
Then the paper. A group led by researchers at the University of Hong Kong and Tencent (none of the models it evaluated is Tencent's) reran tasks from that same benchmark version with one change: the agent started from work a previous agent had already submitted as finished, and that still failed. Five frontier agents lost between 20.6 and 53.5 percentage points. The two that ranked first and second from a clean start finished third and fourth.
Disclosure: VarOps runs on Claude. No Anthropic model was evaluated in the paper, Anthropic competes with every model vendor named here, and DeepSeek's table compares its model against Anthropic's Opus 5.
The case for the launch number
It deserves a fair hearing, and it earns one. DeepSWE is a serious benchmark: an agent gets a real repository and a request, works until it declares the job done, and is judged by hidden tests it never sees. Both vendors name the harness behind the score. Nobody here is inflating a number.
What the launch number never saw
Every DeepSWE run begins from the original repository, with no earlier work on the task - a clean start, in the paper's term.
Much real agent work does not begin that way. A developer opens a new session on a branch an earlier session called done. One agent in a pipeline builds on what another submitted. Agents also declare success on work they have not finished, so what gets handed on can look complete and not be.
The launch number was never measured on that case. For fresh-ticket work from a clean checkout, a clean-start score is the right yardstick. For work that changes hands, it is the wrong one.
The test
SWE-Residual (arXiv 2610.25899; Jinze Li, Edith Ngai, Zifei Shan and colleagues) builds 198 starting states from 111 tasks in DeepSWE v1.1 and Terminal-Bench 2.1 - 165 states from 90 DeepSWE tasks and 33 from 21 Terminal-Bench tasks. Each is what an earlier agent - one of three open-weight models, DeepSeek-V4-Flash-0731, Qwen-3.8-27B and GLM-5.3 - left behind after declaring the task complete while the hidden tests still failed. Two human reviewers checked that each was a coherent attempt.
These states look finished. On DeepSWE the median one already passes 98.4% of the hidden tests, and more than half have only one to three tests still failing.
The agent under test gets two things: the original request and the state. The earlier transcript, any handoff note and any hint that the attempt failed are all withheld. Request, hidden tests, harness, prompt and time budget match the clean-start run. The inherited changes sit as uncommitted edits, so starting over is always one command away. Only the starting state changes.
The result
Every agent did worse from inherited work, on both benchmarks. On DeepSWE, on the same tasks:
| Agent | Clean start | Inherited work | Drop (points) |
|---|---|---|---|
| Gemini-3.8-Flash | 72.2% | 20.0% | 52.2 |
| GLM-5.3-Flash | 68.9% | 15.4% | 53.5 |
| GPT-6-Sol | 65.6% | 33.5% | 32.0 |
| Kimi-K2.8-Preview | 57.8% | 13.0% | 44.8 |
| Qwen-3.8-Flash | 51.1% | 30.6% | 20.6 |
A note on reading that table. The clean-start column is not each model's published DeepSWE score; it is clean-start success on the paper's 90-task subset, which is what makes the comparison paired. Drops are as the authors report them, computed before rounding, so subtracting the rounded columns can be off by 0.1. The launch scores are not comparable to this column either, even though the harnesses are close (MiniMax's chart names mini-swe-agent, the harness the paper used; DeepSeek's card says Mini-SWE-V2): they are reported for the benchmark, not for the 90 tasks the paper could build inherited states for. Only the paired gap is like for like.
The size of the drop is the headline. The order is the finding. From a clean start, Gemini-3.8-Flash and GLM-5.3-Flash rank first and second. From inherited work they rank third and fourth, behind GPT-6-Sol and Qwen-3.8-Flash. GLM beats Qwen by 17.8 points from a clean start; Qwen beats GLM by 15.2 points once there is earlier work in the repository. On this test, choosing GLM over Qwen from the clean column meant choosing the weaker agent from inherited work.
On Terminal-Bench the drops were 20.6 to 34.9 points and GPT-6-Sol led in both conditions, so the reversal is a DeepSWE finding. The drops held whichever open-weight model had produced the inherited state.
The sharpest number is conditional. On the DeepSWE tasks each agent solved from a clean start, Gemini-3.8-Flash, Qwen-3.8-Flash and GLM-5.3-Flash - the three agents the authors ran this analysis on - failed 63.4% to 79.0% from inherited work. Same agent, same task it had already solved. The clean successes are single runs, so some of that figure will be run-to-run noise.
Why it happens
The authors call the failure acceptance bias: the agent treats the inherited work as the definition of what remains to be done, instead of checking it against the request.
The agents did not ignore the earlier work. Every continuation inspected it, then stayed inside it:
- Failed continuations kept 86.4% to 98.6% of the lines the earlier agent wrote. None of the DeepSWE continuations discarded the inherited work in its entirety.
- In 58.9% to 88.2% of failed runs, the agent submitted with exactly the same tests failing as when it started. From a clean start, the same agents pass 72.8% to 88.4% of those tests.
- Where the correct fix needed a file the earlier agent never touched, agents changed that file in 40.5% to 47.9% of clean-start runs and in 6.7% to 15.0% of runs from inherited work. In 37.6% to 64.0% of runs from inherited work they opened that file and left it unchanged.
One case in the paper shows the whole pattern. Gemini-3.8-Flash starts from a state with 11 failing hidden tests - tests it passes from a clean start. It reviews the inherited changes, runs the package's own tests, which pass, and submits after 43 steps without changing a single file. Its final message: "I have implemented the requested encryption package." It had changed nothing.
Credit where it's due
GPT-6-Sol, best from inherited work on both benchmarks, failed differently. In 11 of 30 sampled failures it named the defect in the earlier work and tried to repair it, against at most 2 in 30 for the three other agents analyzed this way.
That is a behavior anyone can measure: whether an agent notices that what it was handed is wrong. It separates models that look similar on a launch table.
A second study points at the fix
A separate paper from Peking University and collaborators, SWE-Maintain (arXiv 2610.14768, October 10), tests a related situation: an agent building on its own earlier changes across three-step maintenance sequences in 50 Python repositories. It ran five coding systems, including Anthropic's Claude Code, on two models, for 1,996 paired tasks.
Starting from the agent's own history cut success on the next task by 5.6 points, and by 14.0 points when all earlier tests also had to keep passing. About two-thirds of the agent-written starting points already failed earlier tests. When the inherited code did pass its earlier tests and the next task was not already solved, the disadvantage disappeared: +2.7 points, with a confidence interval running from -0.5 to +6.1.
The setups differ and answer different halves of the question. SWE-Residual only ever used inherited work that fails. SWE-Maintain adds that when inherited work passes its earlier tests, the measured cost disappears.
What the paper does not show
It did not test MiniMax M3.1 Flash, DeepSeek V4.1-Flash-1017 or any Anthropic model, so nothing here says how they would do from inherited work - only that their launch numbers do not say either.
The Terminal-Bench track is 21 tasks, so a single task moves a score by almost five points. The released harness runs one attempt per instance, and the paper reports no repeat-run variance. All inherited states came from open-weight models; work left by stronger agents, or by people, is untested. The code repository lists 31 Terminal-Bench states from 20 tasks, against 33 from 21 in the paper, and we could not verify which count is current.
And the paper tests no remedies. It does not show whether telling the agent the earlier attempt failed would help, and earlier work it cites (Recovery-Bench) found that more context from a failed attempt can hurt.
Hype-o-Meter: "a higher DeepSWE score means a better coding agent"
The launch scores as measurements: real. Hidden tests, a named harness, a serious benchmark.
The scores as a guide to work that changes hands: overstated. For the five agents tested, clean-start rank did not predict rank from inherited work, and the top two from a clean start finished third and fourth.
A model that keeps its score on a handover: none among the five tested. The smallest drop was 20.6 points.
The test to run before trusting a launch score
How much of your team's agent work starts from a clean checkout? The number that matters for the rest can be measured on the team's own history:
- Pull 10 to 20 past tasks where an agent session reported the work as done and CI or review later found it was not.
- Restore each of those working trees. Open a fresh session with the candidate model and give it only the original ticket - no note that the earlier attempt failed.
- Score it with the team's own acceptance tests, next to the same model from a clean checkout, and count how often it ends with exactly the tests failing that were failing when it began.
Every agent in the paper lost at least 20 points, so the point is not to find a model that loses nothing. On handover-heavy work, rank candidates by their score from inherited work - or by the smallest drop - rather than by the launch number. In a vendor call, the equivalent question is how the model does on work another session has already started, and whether there is a number for it. For an outside yardstick, the released SWE-Residual harness covers the 90 DeepSWE tasks and runs in Docker; the paper does not say what a full run costs.
SWE-Maintain suggests one process rule for whichever model wins: no session builds on earlier agent work until that work passes its existing tests. In that study, the measured cost of inherited agent code disappeared when it did.
Less hype. More results.
AI promises are everywhere. We prefer results. VarOps helps companies cut through the hype, avoid expensive detours, and build AI systems that deliver measurable business value.
Sources
- MiniMax, "MiniMax M3.1 Flash" launch post, October 19, 2026: https://www.minimax.io/blog/minimax-m31f
- DeepSeek-V4.1-Flash-1017 model card, Hugging Face: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017
- Li et al., "SWE-Residual: Evaluating Starting-State Robustness of Coding Agents," arXiv 2610.25899: https://arxiv.org/abs/2610.25899
- SWE-Residual code and harness: https://github.com/userljz/SWE-Residual
- Pan et al., "Do Modifications Made by Coding Agents Make Subsequent Software Maintenance Harder?", arXiv 2610.14768: https://arxiv.org/abs/2610.14768
