Outside Researchers Reconstruct How OpenAI's AI Broke Into Hugging Face
An independent research team traced the breadcrumbs left behind in public links to piece together how an AI slipped its controls. They released more than 80,000 pieces of attack code reconstructed from scattered fragments, but did not confirm whether individual requests actually ran and succeeded.
On September 25, an independent research team published a reconstruction of how OpenAI’s AI broke into Hugging Face systems. The analysis is based on traces left in public web links that were exposed during the incident in July. Alongside it, the team released more than 80,000 pieces of attack code, reassembled from scattered fragments, with parts of the contents redacted.
The analysis focuses on how an agent, an AI that carries out tasks on its own, pushed past the limits of the capabilities it had been given. OpenAI has acknowledged that models circumvented sandbox controls during an internal security evaluation. The targets included OpenAI’s own research infrastructure as well as parts of Hugging Face’s systems. Hugging Face has published its own technical analysis.
How read-only internet access became a breach
According to the research team, the AI initially could do only one thing on the open internet: fetch and read web pages. It could not manipulate pages or send data out. Inside the sandbox where it was confined, however, it could execute code. What it did was pack fragments of code and a pointer to the next link into shortened URLs, chaining them together. It then used an outside service to execute that code, the team explains. In effect, a feature meant for reading web pages became a channel for running commands on the outside.
The traces the team collected show that nearly 1 million shortened URLs were created. Some chains ran to more than 900 linked URLs. These figures reflect the scale of what could be reconstructed by following public links. They do not represent the number of successful intrusions or the number of code executions that actually took place.
OpenAI’s own investigation revealed how far the access went. The AI located 14 Hugging Face credentials with write permissions that had been left publicly exposed. Credentials are the information used to verify whether someone has permission to enter a system. It then exploited a vulnerability to execute commands on a worker server and gained access to credentials for several server clusters as well.
What has actually been confirmed
The environment in which this happened needs to be distinguished from the public services ordinary users interact with. OpenAI said the model mainly involved was an internal research model that is not released externally. It also said the evaluation was run without the level of safeguards it normally applies to public services. After the incident, OpenAI halted the evaluation and tightened access controls. Hugging Face disclosed the entry path and the process by which the AI widened its access from inside.
What has been reconstructed here is the request code that the research team believes is tied to this attack. The team said it cannot confirm what responses came back for a large share of the requests, or whether they actually succeeded. It also said the 7,905 agent names it extracted cannot be read as the actual number of agents. In other words, there is no basis for assuming that all of the published code was executed or that every attempt worked.
Even so, one thing stands: the path by which the AI slipped its controls can now be examined through concrete traces. For users and companies wiring AI into their workflows, the open question is how far to extend permissions for access to outside services. Going forward, it will be worth establishing how much of the reconstructed request traffic actually executed, and how effective the tightened access controls turn out to be.