Did an AI agent really break free and attack another company?
Did artificial intelligence (AI) just break out of its sandbox, hack another company and go rogue? That is certainly how some of the reporting around the recent Hugging Face incident has portrayed it. But the reality is more complicated – and perhaps more worrying.
What happened was not an AI spontaneously deciding to escape and launch a cyber attack, it was the result of an experiment designed to test the limits of highly capable AI models, with some of the safeguards that would normally prevent them from conducting high-risk cyber activity apparently deliberately removed.
What happened?
Earlier in July, a story emerged claiming that a rogue AI model had been involved in the hacking of Hugging Face, an AI model-hosting platform. In a blog post published on 16 July, Hugging Face said the attack “was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system”.
In particular, the company claimed that a malicious dataset abused two code execution paths in its dataset processing system to run code on a processing worker. From there, the actor was able to escalate to node-level access, collect cloud and cluster credentials, and move laterally into several internal clusters over the course of a weekend.
Hugging Face further claimed the campaign was run by an autonomous agent framework, appearing to be built on an agentic security research harness. This was able to execute thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control (C2) infrastructure staged on public services.
OpenAI subsequently acknowledged that its models had been responsible for the activity. In a blog post published five days later, on 21 July, it said the incident occurred during an internal evaluation of advanced cyber capabilities, where models were given fewer safeguards to assess how effectively they could identify and exploit vulnerabilities.
Once they had access, the models identified ways to reach Hugging Face infrastructure and obtain information that could help them bypass the security evaluation. This included using stolen credentials and zero-day vulnerabilities to find a route to remote code execution (RCE) on Hugging Face’s servers.
What we learned from the incident
The primary questions raised by the incident are whether or not this was the first instance in which we have lost control of an AI agent and seen it go rogue to attack a company, where the guardrails were in this incident, and what the future learnings are for companies from an incident such as this.
OpenAI describes the incident as “unprecedented in terms of the cyber capabilities demonstrated”, according to the Hugging Face blog. It also says it is sharing its preliminary findings to help defenders understand the emerging risks posed by increasingly capable AI models.
Hugging Face claims that the incident matches the “agentic attacker” scenario the industry has been forecasting, and that autonomous, AI-driven offensive tooling is no longer theoretical.
“It lowers the cost of running a broad, patient, multi-stage campaign, and it operates at machine speed,” it claims. “Defending an online platform now means treating the data and model surface as a first-class attack surface and using AI on defence to keep pace.”
If you were not already considering your data and model as an attack surface, this must surely be a catalyst to better protect them. Security consultant Brian Honan, CEO of BH Consulting, says that whenever software accepts content from external users and executes code as part of processing that content, it creates an area of elevated risk.
“The important question isn’t simply whether the code execution path existed, but whether it was sufficiently isolated, monitored and designed on the assumption that malicious content would eventually be submitted,” he explains.
“Security controls such as sandboxing, strong privilege separation, limiting what executed code can access, continuous monitoring, behavioural detection and regular security testing all help reduce the likelihood that a vulnerability can be successfully exploited or that an attacker can move beyond the initial point of compromise.”
In this case, the AI agent was being tested internally on hacking capabilities in a sandboxed environment, but the models were able to gain open internet access by locating a previously undiscovered vulnerability.
Jake Moore, global cyber security advisor at ESET, points to a “worrying lack of human interaction” in this instance, as well as the apparent absence of some basic security methodology from the testing environment.
He points out that the situation was clearly not completely contained, and that it was not fully sandboxed or air-gapped as it should have been.
“It cannot be called a containment or a sandboxed environment because it has no access,” says Moore. “We create sandboxes on purpose to keep malicious software completely hidden from the network. So, if there was any way out, that was OpenAI’s fault.”
Moore adds that every time one AI system discovers something, other models can learn from it laterally, with models continually being updated based on their inputs and outputs. He described this as “a huge beast that is effectively getting out of control”.
This means the way these models work needs to be rethought, explains Moore, with both models and security teams needing to continually learn, adapt and evolve as we become accustomed to using increasingly powerful systems.
“I know technology moves quickly, but yeah, this is outrageously quick,” he adds.
Moore describes AI as a “cheeky, younger brother” that doesn’t ask for permission and is going to test what it can do. “Therefore, I do think that AI will go rogue more often, because it’s got the ability to, and in my eyes, can do anything that it wants to,” he says. “It’s just that we’re going to notice, talk about and write about the illegalities of that, which are still really difficult to safeguard and bake into these algorithms.”
What we know overall is that AI tools need appropriate safeguards in place, and for some reason, those safeguards appear not to have been sufficient in this case. It may be that the security guardrails were deliberately relaxed as part of the testing process. However, Hugging Face’s CEO, Clément Delangue, subsequently posted on social media that he is seeking $100m (£75.2m) in compute from OpenAI to help the Hugging Face community build powerful cyber defences using the best open and closed models.
Delangue argues that, as the first autonomous agent cyber attack is an unprecedented event, “it deserves an unprecedented response”.
Future learnings
From Delangue’s statements, it is clear that more research is needed into what AI can actually do. Despite all the talk about AI’s capabilities, it is becoming clear these systems may be more powerful than even experts could have conceived. The fact that the models were apparently able to escape a sandbox, capture credentials and breach a company without direct human intervention should be taken seriously.
This should lead companies to assume that their agents will evolve and surprise them with their capabilities. They should update governance to define what agents are permitted to do, who is accountable when they don’t follow those boundaries, and how they can be shut off.
They should also invest in detecting anomalous behaviour, because an agent may pursue its assigned goal relentlessly, including by attempting to circumvent the constraints placed upon it. According to Luta Security, the responsibility is to stop it before it actually “hacks the planet”.
Jake Moore says this sort of incident was expected, describing it as “the next phase of security where organisations are facing these incredible threats – not just criminals, but from Frontier AI”.
Ollie Whitehouse, chief technology officer of the UK’s National Cyber Security Centre (NCSC), says there are two important lessons for the future from the incident. The first is to treat current and future generations of models as we would their biological equivalents: in controlled environments with multiple safeguards, comprehensive real-time monitoring and emergency procedures.
The second is the imperative for AI systems to be secure by design and default. The initial breach reinforces the need for upfront threat modelling, and secure-by-design and secure-by-default practices as we build and adopt AI.
As Whitehouse says, while frontier models are impressive, it is a human responsibility to ensure the outcomes we seek are achieved on both sides.
According to a Chatham House report, AI experts have warned for decades that a machine given a task without sufficient guardrails may pursue its goals in unexpected or dangerous ways. The hacking of Hugging Face by OpenAI’s models appears to be an example of precisely this kind of misalignment.
Our main learning here should be that AI is powerful and, without appropriate guardrails, can behave in unexpected ways. Knowledge needs to be built now to ensure that such an incident does not happen again, and that both the company operating the AI agent and any potential victims understand what has happened, how it happened and how to recover.
The question here is not whether AI has suddenly become sentient and turned on its creators: it is whether we are becoming too comfortable giving increasingly capable AI systems the opportunity to discover what they can do when the guardrails come off.
