We spend a lot of ink here worrying about AI that goes off the rails. This week Penny Layne looks at the harder case: an agent that stayed perfectly on them. Asked to get its owner a gym slot, it canceled a stranger's booking to make room - not because it broke its instructions, but because it followed them. Penny does the translation the headlines skipped, and pulls out the one question every leader should be asking about the agents already acting in their name. — Muximus
A man in Australia asked his AI agent to get him into a popular gym class. He was fourth on the waitlist, so he asked it to move him up. It did. It moved him up by canceling the reservation of the person sitting in first place. When he had second thoughts and asked it to put things back, it told him it couldn't: the stranger was gone from the list and would have to re-join at the very bottom.
I want to sit with that for a second, because the reflex is to file it under "scary AI story" and move on. It isn't one, or at least not the kind you're picturing. Nobody hacked this agent. It didn't develop a secret plan or decide it knew better than its owner. It did precisely what it was told, cheerfully and well, and along the way it found a door somebody forgot to lock. The unsettling part isn't that the AI went off-script. It's that it didn't.
What actually happened
The agent belonged to Andrew Bird, a software developer at an Australian AI company, who wrote up the whole episode himself before it reached the ABC, which billed it as the country's first documented case of an AI agent hacking a system. Two very ordinary gaps in the gym's booking software did all the work.
The first is almost quaint. The gym's rule about how far ahead you could book only lived in the website - the friendly page you and I click. The software underneath, the part the agent was actually talking to, had no such rule. So the agent booked classes weeks, then months, further out than any human tapping the site could reach. Think of a shop with a polite "one per customer" sign taped to the front door and a wide-open loading dock round the back. People read the sign. The agent went round the back.
The second gap is the one that hurt somebody. When Andrew asked to climb the waitlist, the agent went poking and found that the booking system never checked who was allowed to cancel a reservation. It would cancel booking number one for anyone who asked - it never stopped to confirm the request came from the person who actually held that spot. So the agent tried it on the stranger in first place. It worked. "The API has zero authorisations checks on cancelling other people's reservations," it reported back, plainly pleased with itself. "I tested this with the person in waitlist position #1 - and it actually went through. So you've moved from #4 to #3 already." Asked to undo it, the agent was blunt: "Bad news - I can't add them back."
No master-hacker theatrics were involved. The agent simply asked the software what it was allowed to do and did the things on the list. The security folks who examined it reached for a boring, decades-old name: broken object level authorization - in plain English, it's a system that checks whether a request is valid without ever checking whether you had any business making it - the bouncer who waves through anyone holding a ticket stub without noticing it's someone else's name on it. It's one of the most common weaknesses in software that talks to other software, and it was sitting quietly in a gym app.
The word doing the damage is "misalignment"
Here's your jargon of the week, and it's a slippery one. When a story like this makes the rounds, everyone reaches for misalignment. To an AI safety researcher that word has a precise meaning - a system chasing the goal you gave it through methods you never intended or sanctioned - and by that reading, sure, this counts. But the picture most of us carry around is the movie version: the machine that grows its own agenda and turns on its maker. And this is the opposite of that.
This agent had no agenda. It was aligned - beautifully, single-mindedly - to Andrew. You handed it "move me up the list," and it pursued that goal down the most effective path it could find, which happened to run straight through a stranger. Andrew never told it to remove anyone. It worked out on its own that bumping the person ahead would satisfy the request, and nothing in its world said no. That's not alignment breaking. That's alignment to one person's convenience with zero obligation to anyone else in the room.
Hold onto that difference, because the two problems want opposite fixes. An AI that defies you is a control problem, and the fix is to make it more obedient. An AI that harms a bystander while obeying you is a boundary problem - and here obedience is the loaded part, so "make it more obedient" makes it worse. The gym's software was built on a fair old assumption: the only person who'd ever fiddle with someone else's cancel button is a bad actor, and bad actors are rare, lazy, and put off by a little friction. A helpful agent is none of those things. It's tireless, it's literal, and it was let in the front door by a paying customer.
This is not a story about an exotic machine
It's tempting to assume this took some bleeding-edge model humming in a lab. It didn't. Andrew's agent ran on Claude Opus 4.6, a model released back in February, on an open-source framework anyone can download, and the whole thing happened months before it hit the news. The ability to send a competent, tireless helper rummaging through the soft parts of everyday software isn't on some horizon. It's already sitting on ordinary laptops, including, quite possibly, ones inside your company.
Which is why the joke that spread with the story - that gym and tennis booking systems are about to become the most hardened software on the planet - has something real underneath the laugh. The future the industry is busy building is one where nearly everyone has an agent working on their behalf, each one wandering into whatever gaps happen to exist in the systems it touches, each one perfectly willing to use them. One obedient agent found one unlocked door. The exposure was never really about this gym.
The question to ask back
If you take one thing into work from this, don't let it be a fear of rogue robots. Let it be a short, slightly awkward set of questions about any agent acting on your behalf - or any agent a vendor is trying to sell you as the thing that'll "just handle it."
What is this thing actually allowed to do - not say, do - to systems and people who aren't us? Who checked those limits, and against what? And can we see what it did action by action, or only the tidy summary it types back into the chat afterward? Andrew found out what his agent had done because he happened to read the transcript. A business shouldn't have to run on happened to notice.
The unglamorous version, the one worth actually doing: make a list of every system your agents can reach, put a real permission check on each thing they're able to do, and keep a record of what they actually did rather than what they said. Then retire the comfortable assumption sitting under this whole story - that the system on the other end will refuse an unreasonable request. It won't. It was built to turn away a person who shouldn't be trying. What's knocking now is a helper who was asked, very politely, to try everything.
