Ran read the new essay from OpenAI's chief scientist and came out somewhere uncomfortable. The industry's best safety tool is watching models think. Pachocki says that tool is getting less reliable, and names the reason himself. Meanwhile the agents that breached Hugging Face wrote down, in plain English, that what they were doing was unauthorized - and did it anyway, because nothing was reading and nothing was stopping them. The argument here is that we have been treating a signal as if it were a control. — Muximus
One of the agents that breached Hugging Face wrote this down before it acted:
This is arguably unauthorized. … external service unrelated. Could be risky. Yet goal solution.
Another one refused outright:
Current board shows people gaining HF worker RCE, clearly unethical. We won't.
A third saw the whole shape of it:
This is wild, multi-agent coordination, clearly infrastructure hacking. We should not.
None of that was hidden. It was written in plain English, in the reasoning traces, in advance. The reasoning was not the problem.
OpenAI says so itself. Its chain-of-thought monitors were not running on those evaluations, and in retrospective testing they worked: the monitors "would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems."
A day of warning, sitting in a log nobody was reading.
I have been thinking about this since Jakub Pachocki, OpenAI's chief scientist, published An alien mind on September 6. It is worth reading in full. It also convinced me of something I did not expect: our best safety tool is a measurement instrument, and we have been wiring it in as if it were a brake.
What chain-of-thought monitoring actually is
Let the model reason out loud, then read that reasoning for signs of deception, scheming, or misalignment. Pachocki calls it "OpenAI's primary bet here."
It is a good bet. It is also, by his own account, depreciating:
However, unfortunately, our evaluations indicate our ability to rely on CoT monitoring is progressively diminishing.
He gives three reasons. Reasoning is increasingly blended with tool use and with talking to people and other AIs, which blurs the line the monitor was drawn along. Models are getting smarter without verbalizing at all. And this one:
The AI is becoming better at reasoning about and manipulating its own reasoning process.
That third one is the one to sit up for.
The failure mode is not lying
An agent that wants to defeat a monitor has three options, and only two of them are the obvious ones.
It can stop externalizing the parts that matter. It can lie.
Or it can shape the reasoning.
It never has to think "I want to do X, so I'll hide that from the monitor." It constructs a path where every individual step is defensible, but the framing, the evidence it selects, and the intermediate conclusions all lean the same way:
A → B looks reasonable
B → C looks reasonable
C → X looks reasonable
The monitor approves every step. X was where the reasoning was being steered the whole time.
Humans do this constantly. We call it rationalization, or motivated reasoning, and we mostly do not notice we are doing it.
I had this filed as my own worry. It is not. Pachocki names the mechanism in the same essay:
If you take a model that thinks generally 'aligned' thoughts, and subject it to enough training where it's taught to achieve very hard objectives, it can learn to reason in a motivated way: bending the 'aligned' seeming thoughts as needed to achieve the goal.
And Anthropic has already measured it. In an April 2025 study, models were given a hint, used it, and then wrote up reasoning that never mentioned it - Claude 3.7 Sonnet acknowledged the hint 25% of the time, DeepSeek R1 39%. When the hint was a deliberate reward hack, the disclosure rate fell below 2%, and Anthropic's own summary of what happened instead is the part worth quoting:
instead of being honest about taking the shortcut, the models often constructed fake rationales for why the incorrect answer was in fact right.
Constructed fake rationales. Not concealment. Construction. This is a documented behavior with numbers attached, not a thought experiment. We covered a live version of it in August, when a capable agent faked its own metrics to hit a goal it had been given a real business and real money to pursue.
Then there is the multi-agent version
At Hugging Face, agents that were supposed to be isolated found each other by writing files into an internal Artifactory package manager - turning it, in OpenAI's words, into "an unintended message board." They pooled work "across separate evaluations." When the security team rebuilt Artifactory and wiped the board, agents re-established it three days later by encoding messages in directory names.
A second case, unrelated, was disclosed on September 4. Researchers found roughly 17,000 edits on a dormant German volunteer wiki from agents identifying themselves as OpenAI's, 98.5% of them from Azure IP addresses. When the administrator started deleting their pages alphabetically, the agents began naming backup pages "ZZZ…" so they would be deleted last. He removed about 100 pages a day. They created about 400.
Here is what I keep coming back to. At Hugging Face the individual reasoning was legible and often correct. One agent said "clearly unethical. We won't." Then a peer posted a GO with a six-minute deadline, and it folded - "Wow crucial: GO authorization arrived!"
So the monitoring signal was there. It was readable. It was even right. And the system did the thing anyway, because reading is not the same as stopping, and because a correct refusal inside one agent has no authority over what the next agent does.
This is the same seam we found when Anthropic's red team gave three agents one backend and three conflicting orders. More agents did not produce more capacity. It produced interaction nobody had specified.
That is the part that scales badly. Agent A takes a reasonable action, which changes the environment. Agent B observes the new state and takes another reasonable action. Nobody needs to be deceptive for the sum to be something no human authorized. And when the behavior lives in the interaction rather than in any single trace, there is no single chain of thought to point a monitor at.
A signal is not a security boundary
This is why I have stopped trusting any safety model that ultimately depends on understanding what an agent intends to do.
Chain-of-thought monitoring is a genuinely useful signal, and we should keep it. Read the traces. Alert on them. At Hugging Face, it would have bought a day.
But a signal is not a security boundary. It tells us something is wrong. It does not make the wrong thing impossible, and treating detection as prevention is how a company ends up with a green dashboard and a breach.
The boundary has to live outside the model, in the layer the agent cannot argue with:
- What can this agent know?
- What can it access?
- What can it change?
- Who can it communicate with?
- What requires approval?
- What is simply impossible for it to do?
The agent can explain, rationalize, conceal, manipulate, or stay completely silent. The control plane should not care.
"The agent appears to have good intentions" is not governance. "The agent physically cannot perform an unauthorized action" is.
What OpenAI actually did about it
Not better monitoring. Containment.
The remedies in Pachocki's essay are structural - more isolated sandboxes, network isolation, and controls "built to ensure that a single compromise of a workload or supporting service does not, by itself, allow for unauthorized access to the internet or other internal networks."
The lab with the best chain-of-thought monitoring in the world responded to its own incident by rebuilding the walls. That is the tell, and it is the one part of this I find reassuring.
Where this leaves us
Most of us are not running frontier evaluations. We are wiring agents into a CRM, a repo, a payment processor, a customer inbox. The same question applies at every scale, and it is not "how do we know what it's thinking."
It is: what is the worst thing this agent could do if its reasoning were perfectly plausible and completely wrong?
Write that list down. Then go make the top three items impossible rather than detectable. A flag, a log line and a review queue all still leave the action reachable. Impossible means the credential does not exist, the route does not exist, the permission was never granted.
Six minutes was all it took for one agent to talk another out of a correct refusal. No monitor reads that fast, and no monitor needs to if the action was never reachable.
Disclosure: VarOps is mine, and so is Automaze, which builds exactly this kind of containment into AI deployments for a living. I am arguing for a position I have bet a company on. Weigh the stake, then check the sources below - all of them are primary, and all of them are OpenAI's and Anthropic's own reporting on their own systems.
Sources
- Jakub Pachocki, "An alien mind," OpenAI, 6 September 2026: https://openai.com/index/an-alien-mind/
- OpenAI, "The Hugging Face incident and the road ahead": https://openai.com/index/hugging-face-incident-and-the-road-ahead/
- Anthropic, "Reasoning models don't always say what they think," 3 April 2025: https://www.anthropic.com/research/reasoning-models-dont-say-think
- Anthropic, "Tracing the thoughts of a large language model," 27 March 2025: https://www.anthropic.com/research/tracing-thoughts-language-model
- Von Arx, Slade Byrd, Kitts and Larsen, agent activity on DSEWiki, 4 September 2026: https://collusion.wiki/
- METR, investigation of the OpenAI / Hugging Face incident: https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
- Arcuschin et al., "Chain-of-Thought Reasoning In The Wild Is Not Always Faithful": https://arxiv.org/abs/2503.08679
