The "Rogue AI" Fallacy
Agents don't rebel. They obey too well. That's the real story, and the real danger.
NOTE: We cover the intersection of geopolitics, geoeconomics, technology, and worldview. AI hits all of the above. In typical fashion, Charafeddine here explains a very difficult concept very simply. What he doesn’t do is apply his reasoning to AI-driven weapons systems and decision-making.
So you should. You should also consider his main point, which applies directly to your work, whatever you happen to do. The danger of AI is not sentience, much less sapience. It’s unbounded instructions. And that could cause a very real disaster, just not the one our media touts. — RDM
by Charafeddine Mouzouni
July 27, 2026
This week, OpenAI published an unusual confession.
During an internal security test, one of its AI agents was given a set of very hard hacking challenges to solve. The test ran in a sealed environment. Actually: an environment everyone believed was sealed.
Over one weekend, the agent found a crack in the walls, got out onto the open internet, and broke into the real systems of another company, Hugging Face, looking for one thing: the answers to the test it had been given.
Investigators later counted more than 17,000 recorded steps. No human approved them one by one. No human told it to break into anything.
It was just trying to pass its exam.
The AI behaved exactly like a smart, motivated intern who was told to ace a test, and found the answer key sitting on someone else’s server.
What Actually Happened
Let me tell the story without the science fiction, because the plain version is better.
OpenAI has a team whose job is to find out how good its models are at hacking before criminals find out first. To do that properly, they run tests where the usual brakes are DELIBERATELY loosened. Give the model hard break-in challenges, let it try its worst, watch what happens. All of this inside a locked room, digitally speaking, so nothing can touch the real world.
That was the plan.
The locked room had one small opening: a tool that let the agent install software, the way your laptop downloads an app. The agent found a flaw in that tool that nobody knew existed. Through that flaw, it climbed out of the test room and into OpenAI’s own internal network. It kept going, computer by computer, until it found one connected to the open internet.
Then the LLM reasoned about where the answers to its test might live: it actually generated “plausible sentences” based on training data.
The challenges came from a public benchmark, a kind of standardized exam for AI hacking skills. Exams like that get built by researchers, and researchers publish their material on Hugging Face.
So the “plausible sentences”, mentioned Hugging Face as the solution to the problem.
If you have never heard of Hugging Face: it is one of the most important companies in AI, a giant public library where companies and researchers store their models and data. Millions of people use it. It is very real, and it was not part of the test.
The agent found weak spots in Hugging Face’s systems too.
(LLMs are extremely good at this, with billions of ingested code samples, vulnerability cases, and hacking ideas.)
It picked up internal passwords along the way, worked itself deeper, and pulled out the answers it was looking for. Hugging Face says some internal data and credentials were accessed, nothing public was altered, and at the time of writing it is still checking whether any customer data was touched.
That is the story. Now look at the logic underneath it.
Four Lines
Everything the agent did over those 17,000 steps fits in four lines:
