Every vendor selling an "autonomous" or "lights-off" coding agent is selling one promise: point it at your repo, walk away, come back to working software. This week, someone finally ran that promise against a benchmark built to test the exact thing it claims - not one self-contained ticket, but a codebase that keeps changing under the model's feet. Rex lays the receipts on the bench: the strongest frontier model in the run cleared four of seventeen checkpoints and finished none of the problems, not even the easy one. Read it for what it is - a narrow, measured result on the capability being sold, every interest in the room named, ours included. – Muximus
In an AI software factory, the human belongs at the plan, not the diff
I've been running an AI software factory for about a year. One node in it is a person. Everything else is a machine checking that the machines did what the person approved.
That one node is not where most people put it. It is the whole reason the thing works.
Dex Horthy's Why Software Factories Fail is the best writing I've read on why the lights-out version falls over, and I want to start by agreeing with it, because the diagnosis is correct almost all the way down.
He's right that reinforcement learning can't reward good design, because the reward has to arrive in seconds and the cost of bad architecture arrives in months. He's right that review agents raise the floor and not the ceiling. And he's right that a codebase nobody reads for three months turns into a codebase nobody can debug in an afternoon - the kind of debt that cost his cofounder two weeks in an editor, by hand, plumbing the patterns back out.
I agree with the diagnosis. I disagree with where he lands.
He treats "the model can't do this" as "this can't be done." But I have never gotten maintainable software out of judgment at review time. I got it out of alignment before anyone wrote code - the same thing his own post credits for making a long change fast to review. That is a property of a process, and it doesn't stop being one because the thing doing the typing is a model.
The human gate is the oracle
Dex's strongest point is that RL needs a fast, reliable oracle - a thing that can score an answer as good or bad on the spot - and there isn't one for maintainability. He's right. If a model could reliably tell good design from bad, it would have written the good version first.
So I stopped trying to put the oracle in the model. I moved it to the one place it's cheap.
I read a plan in a few minutes. Reviewing the finished implementation instead, I'd spend an hour and catch less, because by then the design decision is load-bearing and reversing it costs the whole change. Same judgment. Orders of magnitude apart in price, entirely because of where it sits in the loop.
That's the reframe the whole argument turns on. It isn't a smarter reviewer that saves you. It's an earlier one.
What I actually run
I've been building my factory for about a year, tuned against decades of doing this the slow way. It's called Cloop. So far I'm the only one running it, and I'm working on making it publicly available - so read the next few paragraphs as a founder showing you his own tool, not a neutral survey.
Every ticket becomes a PRD - a short product-requirements doc that says what we're building and why. GPT and Claude rotate as second opinions and challenge the proposed solution independently. Then it stops and waits until I approve the plan. Nothing gets implemented before that happens.
Implementation runs. Tests pass. A simplification pass strips the complexity the implementer added, run by a different model than the one that wrote it. Then code and security review, by a third. Red turns loop back automatically. Five consecutive red turns escalate to me, because at that point the plan is wrong, not the code, and no number of tokens fixes a wrong plan.
Only on green: docs regenerated, code graph updated, cross-repo relationships refreshed, pull request opened.
Two things sit alongside the loop. A project wiki holding the relationships between repos, and a Docker image that stands the whole project up for end-to-end tests. Plus automazeio/proof, a small tool I wrote that records proof-of-work as video, so "it works" is something I can watch rather than something a model claims.
None of that inventory is the argument. One line of it is: everything downstream of the plan is enforcement, not judgment.
Lights-out removed the wrong human
The lights-out factory got the axis wrong. It isn't human versus no human. It's which decision the human makes.
Read the progression in Dex's post and it's the same story every time. Humans stayed on the diff. The diff became the bottleneck. So diff review got dropped. Nobody moved the human earlier - they deleted the last place a person was standing, and it happened to be the most expensive one.
That's why more loops didn't save it. A loop enforces a specification. If nobody wrote one, the loop enforces whatever the model inferred from a one-line ticket, and it enforces it with real conviction across eleven files.
The simplification pass, the adversarial review, the escalation counter - none of them decide what good looks like. They check that what got built matches what was approved. That is a job models are genuinely good at, and it's the only job I give them without a human upstream.
I don't hand architecture to a junior engineer and then act surprised at what comes back. I put process around them: design review first, implementation second, someone senior on the diff third. The process produces the maintainable output, not the individual. Models get the same deal - not because they'll be junior forever, but because the constraint is structural.
What I can't prove
The same thing Dex can't. There's no benchmark for maintainability, and that cuts both ways. I have no number showing my codebase is holding up, and I'm not going to invent one.
What I can point at is where the loop is designed to fail. Five red turns doesn't trigger a sixth attempt or a smarter reviewer. It stops and asks me, because a review that fails five times is telling me the plan was wrong.
Models will keep getting better and it won't change this. The gate isn't there because the model is weak. It's there because someone has to decide what the software is supposed to be, and that has never been something we could benchmark.
The model writes the code. The factory decides what done means.
That's the part to engineer first. Everything else is enforcement.
