The OpenAI and Anthropic containment breaches are a bit spooky, but also quite silly
Picture this: you wake up in a blank room with no real memory of who you are or how you got there. Somehow, though, you know what you’re supposed to do – get out and achieve an objective of some sort – but there’s no clear way to do it.
Then, you spot it.
In the corner of the room is a torn-off scrap of paper. It gives exact details of how to escape… and it’s written in your own handwriting.
If you listen to some of the rumors following a ChatGPT agent’s breach of containment and attack on Hugging Face, you’d be forgiven for thinking this is what happened with OpenAI’s generative AI software.
According to Reuters: “In one case, an agent left notes apparently for future versions of itself, according to three people familiar with the matter. The notes, found in a part of OpenAI’s infrastructure, laid out instructions for how agents could free themselves from OpenAI’s internal constraints.”
This does all feel a little bit unnerving: an AI that has escaped onto the open internet and is determined not to be restrained. It’s hard not to anthropomorphize the software or give it agency – decades of sci-fi novels and films have primed us for this very occurrence.
Let’s rewind, though, and see if some of this spookiness rubs off.
There’s no agency in the agent
Apparently there’s one thing that can never be said enough: AI, whether agentic or not, can only act within the parameters that have been set. It doesn’t have consciousness, agency, or will.
Both the ChatGPT agent and Claude agent – which we will come to in a moment – that breached the sandbox they were supposed to be contained in were doing what they had been directed to do.
One of the things agents can be enabled to do is update – but not completely rewrite – a file called agents.md. When the “people” cited by Reuters talk about the ChatGPT agent leaving notes for itself on how to escape in the future, it’s entirely possible that the software was running as intended – that updating its agents.md file as the test progressed was part of the task.
Framed like this, the “unprecedented cyber incident, involving state-of-the-art cyber capabilities”, as OpenAI termed the Hugging Face incident, feels a lot more like a piece of software following its programming.
The lessons for businesses
For OpenAI, Anthropic, various pundits, and IT and cybersecurity professionals, the lesson to be drawn from all of this is to be very afraid and prepare to face powerful cyber attacks ‘at the speed of AI’.
For me, though, the real message businesses should take is this: make sure you’ve set your parameters correctly. Check them, then check them again. And not just once, but on a regular basis, as monitoring post-deployment really is key.
Both OpenAI and Anthropic were working with partners to test their models, but in the case of the latter it does seem that the ‘escape’ wasn’t as a result of a complex chain of attacks or a zero-day vulnerability. Instead, there was a “miscommunication”, which led to three instances of Claude, Mythos, or another unreleased model accidentally being given access to the internet.
A misconfiguration, in other words.
In the case of OpenAI, the model “exploited a previously unknown zero-day vulnerability in Artifactory, a package registry cache proxy” in order to escape from ExploitGym. There’s an argument to be made, perhaps, that an “even more capable pre-release model … with reduced cyber refusals for evaluation purposes” should have been airgapped if it was known to be so powerful and scary. As it is, the company has been forced to clarify: “The pre-release model mentioned in our blog post is an internal-only research prototype and was never intended for public release.” It’s since been deactivated, encrypted, and subject to restricted access.
What, then, of Hugging Face? What lessons can we learn from the named victim in all of this? That’s easy: the importance of effective monitoring.
One of the earliest use-cases for AI technology has been in cybersecurity, where it’s been used for over 10 years. In its own blog, Hugging Face said: “The attack was initially surfaced through AI-assisted detection. Our anomaly-detection pipeline uses LLM-based triage over security telemetry to separate real signals from the daily noise, and it was the correlation of those signals that flagged the compromise.”
The company offered its own advice, too, based on its experience: “The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment.”
In other words, be ready for the attack because it’s a matter of ‘when’ not ‘if’ it happens – another long-standing cyber industry mantra but, like ‘check your settings’, it’s still very much worth abiding by.
