> ## Content Index
> Fetch the complete content index at: https://varops.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# DoorDash’s AI agents cleared 45 of 50 stale feature flags at an average of $4.79 each
- URL: https://varops.com/doordashs-ai-agents-cleared-45-of-50-stale-feature-flags-at-an-average-of-4-79-each/
- Published: 2026-11-02T13:09:48.000Z
- Updated: 2026-11-02T13:09:48.000Z
- Description: DoorDash's engineers gave 50 stale feature flags to Claude agents and published every count: 31 clean, 14 after one more round, 5 an engineer had to finish. The method, and where the evaluation stops.
- Author: Rex Factor
- Tags: Proof of Work

*Most of what Rex has filed this autumn has been about the cost of letting agents near code:* [*the patch queue*](https://varops.com/how-to-size-the-patching-work-ai-found-flaws-create-from-bank-of-americas-1-000-extra-contractors/)*,* [*the regressions*](https://varops.com/fixing-a-bug-and-breaking-nothing-are-two-separate-tests-for-ai-written-patches/)*, the review hours. This one is a backlog getting smaller. DoorDash gave its stale feature flags to a pair of Claude models, kept a single human decision at the front of the job, and published the per-flag bill along with the five it could not finish.* [***Rex Factor***](https://varops.com/columnist/rex/) *walks through how they built it, where the evaluation stops, and the three questions that tell a team whether its own backlog is the same kind of job. —* [*Muximus*](https://varops.com/columnist/muximus/)

---

DoorDash gave a set of AI agents *a backlog* and published the scorecard. Of 50 stale feature flags, the agents produced pull requests engineers accepted for 45: 31 on the first try, 14 after one more round, 5 that an engineer had to finish. The average cleanup cost $4.79 in model API fees and took 13.8 minutes, and before any of it*,* an engineer confirmed the value each flag should keep.

By the engineers' own account*,* the backlog grows faster than their teams can clear it. Two design choices carry the result. That confirmation is the one human decision, and it comes before any code changes rather than after. And the agents read the right answer from the system that holds it, rather than inferring it from the code.

## The backlog

A feature flag is a switch in the code that turns a feature on for some users and off for others. Once the feature has fully rolled out, the switch and the code behind its "off" side are dead weight. At DoorDash, flags are called Dynamic Values, and two of its engineers, Atharv Chandratre and Jai Datar, count more than 56,000 of them across about 623 repositories. As of May 2026, 1,659 were stale: untouched for 90 days, still referenced in code, not archived or retired, and not on an exclusion list. About 2,300 new flags are created every month. The internal estimate is one to two hours of engineering time to remove each stale flag by hand; the paper puts the whole backlog at "thousands of engineering hours."

The standard automated tool did not fit. Uber's Piranha, the best-known rule-based cleaner, rewrites code by matching syntax patterns. DoorDash wraps each flag in three layers - a name constant, a dependency-injected wrapper function, and the business logic that calls it - so the flag's name, the call, and the branch sit in different files. Add flags that hold strings or numbers rather than on/off values, and cleanups that span files, and the paper calls Piranha and its successor "structurally inapplicable" there. Even a plain on/off flag, by the paper's account, means coordinated edits across 3 to 20 files.

The evidence for everything below is their paper, posted to arXiv on October 22 and, per the listing, accepted to the industry track of the ICSME 2026 software maintenance conference.

## How they pulled it off

The system is built on Google's Agent Development Kit and runs in two phases.

Phase one is the lookup. An orchestrator agent running Claude Sonnet 4.6 reads the flag's Jira ticket, finds every reference in the code, and asks DoorDash's experimentation platform - through an MCP connection - what the flag is actually set to in production. That step exists because the default in the code can differ from the live value, and a tool that reads only code "could silently hardcode the wrong value." A flag for a feature that was later abandoned shows a 0% rollout, so the agent removes it using the baseline value instead. The orchestrator then writes a short report: the value it intends to keep, and the files it will touch.

An engineer confirms the value. That is the only human step before a pull request exists.

Phase two is the edit. The orchestrator starts one removal agent per flag, running Claude Opus 4.6, up to four at a time per repository. Each works in its own git worktree - a separate checkout of the repository - with a one-hour time limit. It removes the constant, the wrapper and every call site, simplifies the branches, and fixes or deletes the affected tests. It cannot open a pull request until the build and tests pass and the Detekt linter is clean. Its instructions also require tests to cover at least 95% of the changed lines, a check the agent assesses for itself and sometimes gets wrong; DoorDash's CI system enforces the 95% line after the pull request opens. The finished pull request is assigned to the flag's owner.

For teams attempting the same, the authors suggest three practical prerequisites: a reliable source of the deployed flag state, [isolation between parallel agents](https://varops.com/how-to-measure-parallel-agent-interference-on-a-repo-before-buying-a-tool-to-prevent-it/), and deterministic checks before a pull request is opened. Their conclusion adds a fourth, splitting the work between models. The cheaper model orchestrates and the more expensive one edits, which the authors say lowered API cost; they give no figure for the saving.

On isolation they have a failure to show for it. Early runs without separate worktrees had agents editing the same files at once, "producing corrupted diffs."

## By the numbers

The 50 flags were the 50 most recent cleanups the system had performed as of April 14, all in Kotlin repositories. The authors sorted them by difficulty afterward: 6 simple, 18 medium, 26 complex.

| Outcome                       | Flags    |
| ----------------------------- | -------- |
| Accepted first try            | 31 (62%) |
| Accepted after one revision   | 14 (28%) |
| Failed - engineer finished it | 5 (10%)  |

| Difficulty   | Cleaned               | Avg. time | Avg. cost | Median cost |
| ------------ | --------------------- | --------- | --------- | ----------- |
| Simple (6)   | 6 of 6, all first try | 7.5 min   | $2.69     | $2.50       |
| Medium (18)  | 17 of 18              | 10.4 min  | $3.46     | $3.42       |
| Complex (26) | 22 of 26              | 17.7 min  | $6.20     | $4.50       |

Across all 50 the median cost was $3.49 and the median time 12.3 minutes. Complex flags pull the averages up; the worst of them reached about 40 minutes and $19.

Run the whole stale backlog at the average cost and 1,659 flags come to roughly $7,950 in API fees. At DoorDash's estimate, the same work by hand is 1,659 to 3,318 engineer hours. That arithmetic is ours, not the paper's, and it leaves out the engineer's time to confirm each value and [review each pull request](https://varops.com/producing-code-with-ai-agents-and-trusting-it-are-two-separate-costs-heres-where-do-they-go/), which the paper does not measure.

## What went wrong

CI passed on the first attempt for 44 of the 50\. All six misses were test coverage; none were incorrect code. The paper says the agent's own coverage self-assessment occasionally miscalculated, and CI caught the gap. Our read: the check that runs as code did the work the instruction in the prompt could not.

Of the 14 revisions, six were for test coverage. The paper describes the other eight as incomplete dead-code removal, typically a dead variable or a reference deep in a call chain, which the agent fixed when asked again. Its own per-tier breakdown counts them differently - six incorrect test changes and two leftover unused variables - so the paper does not settle the exact split.

The five failures share one cause. In each, the flag's value was threaded through a long chain of method calls across interfaces, and the agent removed most of the dead code but missed one or two links. Flag type and file count did not predict failure. Chain depth did.

The authors report that across all 50 pull requests they "did not observe any bugs or regressions introduced by the changed code," and they conclude that the system's limit is completeness, not correctness. One reviewer's note marks what that test does not cover: the removal was correct, but variable names "should have been updated, since the flag's removal changed their semantics." A post-cleanup review step for that sits on the authors' list of future work.

## What the evaluation cannot show

Several limits matter for anyone using these numbers to plan. The first is our observation; the rest are stated in the paper or follow from what it leaves out.

- DoorDash engineers built the system, reviewed its pull requests and wrote the evaluation. No one outside the company has checked the results.
- The 50 flags were the system's most recent 50 cleanups, not a random draw from the stale backlog. The paper does not say how flags were chosen to be fed to it.
- Each flag was run once. The authors note a re-run could come out differently.
- The difficulty labels were assigned "post hoc," after the cleanups had run.
- The one-to-two-hour manual figure is an internal estimate, not a measurement.
- The paper gives no time window for the "did not observe any bugs or regressions" finding, and no production numbers beyond the 50\. How much of the backlog has been cleared since is not reported.
- It is one company, one flag pattern and one language.

If you are carrying these numbers into a planning meeting, weigh the second point most. Nobody outside DoorDash knows whether the flags given to the system were the easier ones, and a team's own backlog will not arrive pre-sorted.

The paper also notes that Google, Duolingo, Atlassian and others have described using language models for flag cleanup, without publishing success rates or costs.

## The method an operator can copy

What transfers is not feature flags. It is the shape of the job: a backlog of small, repetitive changes, a system of record that already holds the right answer, and a check a machine can run before a person looks.

For a team weighing the same approach on its own backlog, three questions decide whether that backlog is a candidate:

1. Is the correct outcome stored somewhere an agent can query - a flag platform, a schema registry, a dependency manifest - or would the agent have to infer it from the code?
2. Can a deterministic check reject a bad change before a reviewer sees it - build, tests, coverage, linter?
3. Is there a per-item manual estimate to compare against?

Three yeses make it a candidate. Run the agent on 50 items, one isolated checkout each, with a person confirming the one ambiguous decision up front. Count first-try, one-revision, and failed outcomes and the cost per accepted change, then price the rest of the backlog from that. By the paper's survey of industry reports, DoorDash's are the only published success-rate and cost figures for LLM-based flag cleanup, caveats and all: 62% first try, 28% after one revision, 10% failed, and medians of $3.49 and 12.3 minutes per item. A team that lands well short of that should look first at how big the failed changes ran through the call chain.

From thinking to doing. 

[![CTA Image](https://cdn.varops.com/content/images/2026/10/logo-black.jpeg)](https://varops.com/transformation/) 

****From proof of concept to proof of value.**

Proof matters more than promises. VarOps doesn’t stop at recommendations or prototypes. We work with companies to install, operate, and improve AI systems that deliver results in the real world.

[See How We Deliver → ](https://varops.com/transformation/) 

## Sources

- Atharv Chandratre and Jai Datar (DoorDash), "Automating Feature Flag Cleanup at Scale with Multi-Agent LLM Systems: An Industrial Experience at DoorDash," arXiv 2610.31351, October 22, 2026: [https://arxiv.org/abs/2610.31351](https://arxiv.org/abs/2610.31351?ref=varops.com)
- Full text (HTML): [https://arxiv.org/html/2610.31351v1](https://arxiv.org/html/2610.31351v1?ref=varops.com)