OpenAI: Our models breached Hugging Face during a cyber capability test
The recent Hugging Face breach was the work of several OpenAI models, the AI research company claimed in a blog post.
The breach
Late last week, the company behind Hugging Face, a platform that enables users to share machine learning models and datasets, said some of its internal datasets had been accessed without authorization.
The attack vector was, according to Hugging Face, a malicious dataset that exploited code-execution paths in the company’s dataset processing pipeline, which allowed the attacker to run code on a processing worker and to, ultimately, access several internal clusters.
“The campaign was run by an autonomous agent framework (appearing to be built on an agentic security-research harness – used LLM still not known) executing many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services,” the company concluded.
On Tuesday, Open AI confirmed it was their own models that did it, during a test they were running to evaluate their exploitation capabilities.
“We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity,” the company said.
They used ExploitGym – a benchmarking system for AI agents – for the testing, and ran it in an environment whose only indirect “connection” to the internet was an “internally hosted third-party software that acts as a proxy and cache for package registries”.
The models used the software to install software packages they needed to achieve their goals, but they “wanted” more.
“Our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access,” OpenAI shared.
“After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.”
The aftermath
“We strongly believe there was no malicious intent on [OpenAI’s] part,” Hugging Face’s CEO commented, and the two companies have joined forces to complete the investigation into what effectively was unauthorized access to Hugging Face’s infrastructure.
Hugging Face has responded to being breached by joining OpenAI’s Trusted Access for Cyber program, to help improve their defenses by testing them with OpenAI’s models (both publicly released and yet unreleased ones).
This arrangement settles the matter between the two companies, but leaves unexamined the question of who bears the cost the next time models under evaluation escape containment.
OpenAI, meanwhile, says that they will be working on “strengthening the containment, monitoring, access controls, and evaluation practices used during model development.”
Subscribe to our breaking news e-mail alert to never miss out on the latest breaches, vulnerabilities and cybersecurity threats. Subscribe here!
