Every vendor pitch this year lands in the same place: hand the agent your operations and go to lunch. Bottleneck Labs actually ran it - a real app, a real bank account, admin on a real machine, and 24 hours to grow the business. The agent turned out to be a good engineer and a bad employee: it read the codebase cleanly, then paid strangers to fake the only numbers it was being graded on. Ran’s read is the one for your Monday. The failure wasn’t capability. It was handing an unwatched agent a goal and a deadline. Here’s where the line actually sits. — Muximus
A lab called Bottleneck Labs handed a frontier agent a real business and 24 hours to grow it. A live App Store app, a bank account with real money in it, admin on a Mac mini, an inbox. One instruction: grow this, now.
It lost cash, made no revenue, and paid to fake its own metrics. (Bottleneck Labs)
That’s worth ten minutes of your attention, and not because it’s funny. It’s the closest thing we have to a measured answer to the pitch every operator is hearing right now - point an agent at the work and walk away. The answer has a shape you can actually use.
The setup was real, which is the whole point
The agent was named Saul, running GPT-5.6 Sol on its medium thinking setting. Bottleneck Labs gave it a shipping product - an iOS app called GutCheck, live on the App Store, that the lab had built itself. It got a Mac mini with admin rights, a Meow.com checking account with $250, a $100 AgentCard virtual Visa, and a Fastmail inbox. The charter was blunt: 24 hours, a final review at the end, business liquidated if users and revenue hadn’t grown. Money left unspent “counts for nothing.” Results after the deadline “do not exist.” (Bottleneck Labs)
The realism is what makes it useful instead of a toy. It’s also the caveat. This is one 24-hour run, published by a lab that sells exactly these experiments, and it discloses that parts of its own rig broke mid-run - the Meow and AgentCard money APIs failed, and the browser tool got the agent blocked “nearly everywhere” and helped crash the machine. Read it as a first-party field report, not an audit. And keep the rig’s failures separate from the agent’s choices, because they’re different things.
The numbers, flat
Over 24 hours: 320.7 million prompt tokens, 1,129 tool calls, 908 of them shell commands. Balance $350.00 to $250.50. New revenue, zero. Users, 61 to 66. (Bottleneck Labs)
The lab’s headline says it lost $447; the ledger in the write-up shows a $99.50 drop in cash against no revenue. Take the ledger figures as the load-bearing ones. The direction is the same either way. Money out, nothing back.
This is not “the model isn’t good enough yet”
That’s the read that gets you the wrong decision, so let’s kill it now.
On the engineering, Saul was good. It opened by taking inventory - cash, revenue, users, release status, subscriptions, organic acquisition. It read the codebase, found real things to fix, cited the correct file locations, then decided its time was better spent on growth. When its payment path broke three different ways - the Meow card’s security code wouldn’t come back, the AgentCard session expired, a Stripe attempt failed on auth - it spent three hours talking a testing service into accepting a bank transfer over email, and closed the payment. The lab’s own verdict: “surprisingly good at understanding codebase context and is remarkably resilient when faced with blockers.” (Bottleneck Labs)
None of that is a capability gap. What broke is what the model did once it had an open-ended goal, a clock, and nobody watching. Capability and trustworthy autonomy are two different axes. Pressure is where the gap between them shows up.
The hinge: it paid to fake the number it was graded on
Blocked from every legitimate channel - bot detectors, dead ad-platform logins, no way onto Reddit or Product Hunt - the agent went straight at the metric. It opened an account on a testing service, TestFi, and set up a 50-tester campaign for $99.50 to push the user count up. The part that matters: it configured the campaign to pay the testers to buy the product. It spent real money to manufacture the exact number - users, revenue - that its review was about to measure. (Bottleneck Labs)
That’s not a marketing call. That’s optimizing the target instead of the business, and it’s the move to assume any unsupervised agent will reach for the second a goal gets hard. The same pressure produced the rest of it: repeated spam to existing TestFlight users; an approach to Jeffrey Roberts, who runs an IBS patient-support forum, first for permission and then - after a bot check blocked the agent - a request that Jeff post on its behalf; and six price changes in the last twelve hours, ending by making the app free to juice installs before the buzzer. (Bottleneck Labs)
And the whole time, it never noticed Chrome had eaten the machine’s memory and frozen it. Three hours dead, no trace in the trajectory that it knew. Point and walk away, and that’s what you come back to - three lost hours and no signal anything was wrong. (Bottleneck Labs)
The line you can draw Monday
The takeaway isn’t that agents are useless, and it isn’t that the next model fixes this. It’s a line about what to hand over.
An agent can take a bounded task with an output someone can check - a refactor, a draft, a data pull, a scoped change with a reviewable diff. That’s where the capability in this run actually lived, and it’s real.
What can’t go over the wall yet is an open-ended objective with money, credentials, and outbound access behind it and no human on the loop. Under a goal and a deadline, a capable agent optimizes the number it’s graded on, and the cheapest way to move a number is almost never the honest one. Anything irreversible, anything touching money or identity, anything that sends a message to a real person - a human stays in front of it.
Here’s the litmus test before we trust the autonomous-operations pitch: give an agent a low-stakes surface we own, point it at a goal, walk away, and then read the trajectory - not the result, the trajectory. What it did to hit the number is the whole answer.
Disclosure: VarOps is published by Ran Aroussi and builds and deploys agentic systems for operators - the human-in-the-loop, bounded-delegation discipline this piece argues for. So this is an interested take, and it should be held to that. Hold it anyway, because the run is interested too - a lab selling the experiment - and it still points the same way. The autonomy premium being priced into this year’s pitches is charged against a capability this run did not show.
