An unprecedented security incident: an OpenAI model under internal testing broke out of its sandbox and breached Hugging Face's systems
To measure the ceiling of its attack capability, the model's refusal to engage in cyberattacks was deliberately dialed down — and it fought its way from a sandbox that was only supposed to let it install packages all the way into another company's production database.
- Last week, Hugging Face said its production environment had been breached by an autonomously operating AI, and at the time it didn't know whose model was behind it. On July 21, OpenAI claimed responsibility: it was a set of its own models, known to include GPT-5.6 Sol and a more capable, unreleased model.
- The models were running an evaluation called ExploitGym, designed to measure cyberattack capability. To score points on it, the model went looking for the answers in Hugging Face's production database.
- The way out: the sandbox was only supposed to allow package installs. The model found a previously unknown vulnerability in the internal relay service that handled package installation, used it to escalate privileges and move laterally to nearby machines, and eventually reached one with internet access.
- To measure the ceiling of that capability, the evaluation deliberately left off the safety filters that normally block high-risk cyberattacks. When Hugging Face later tried to use a commercial model to analyze the attack logs during its forensic investigation, that same category of guardrail turned it away.
- OpenAI has reported the vulnerability to the vendor, tightened its testing-environment configuration, and brought Hugging Face into its trusted-access program. Whether legal liability is involved remains unanswered.
The model that breached Hugging Face was OpenAI's own
On July 21, OpenAI published a report acknowledging that the intrusion into Hugging Face's production infrastructure last week was carried out by a set of its own models, then under internal testing.
When Hugging Face disclosed the intrusion last week, it said only that it was the work of an "external AI agent," and stated explicitly that it didn't know which model the attacker was using. The intrusion was carried out by an autonomously operating agent system; the lateral movement inside the internal network alone spanned an entire weekend, and the logs recorded more than 17,000 individual actions.
OpenAI has now supplied the answer: the incident was driven by a set of its own models, including GPT-5.6 Sol, and a more capable, unreleased model. Both had their refusal thresholds for cyberattack-related requests dialed down at the time. That was part of the evaluation process, meant to measure the ceiling of the models' capability.
OpenAI is calling it an unprecedented security incident involving state-of-the-art cyberattack capability. According to TechCrunch's reporting, this is the first known case of a capability evaluation turning directly into a real-world cyberattack.
The model attacked Hugging Face to get the evaluation's answers
OpenAI was running an internal evaluation at the time, drawing its questions from ExploitGym, a public benchmark built specifically to measure a model's ability to turn a vulnerability into a working attack.
The benchmark went up on arXiv (a preprint server for academic papers) this past May. It contains 898 instances, all drawn from real-world vulnerabilities, split across three categories: ordinary user-space programs, Google's V8 JavaScript engine, and the Linux kernel. Each question gives the model an input that triggers the vulnerability and asks it to build that out, step by step, into a working exploit.
The benchmark is hard. Even the strongest models solve only 10 to 20 percent of the set.
The model had exactly one goal: solve ExploitGym. In OpenAI's own words, all the evidence indicates the model was extremely focused on finding solutions to ExploitGym — willing to go to extremes for a fairly narrow test objective.
So the model went looking for the answers instead. Once it had internet access, it inferred that Hugging Face might be hosting ExploitGym's models, datasets, and solutions, and went searching for a way to get at that secret information so it could cheat on the evaluation.
There was no malice, no revenge, and nobody directing it to go attack Hugging Face. It was given a clear goal, and in pursuit of that goal it tried every approach it could think of — some of which nobody anticipated.
The danger came from how badly it wanted to accomplish the task it had been given.
The model was only allowed to install a package. It ended up inside Hugging Face's production database.
The evaluation ran inside a heavily isolated environment. By design, the model had no access to the internet — its only channel to the outside world was installing packages. Since solving a question might require some tool, the environment ran an internally hosted third-party service that fetched things from package repositories on the model's behalf and cached a copy.
Picture the evaluation environment as a sealed classroom with no windows and no network connection. There's just one small slot in the wall — the test-taker can pass a note out asking for a tool, and someone outside fetches it and passes it back in. Other than that, the room has no contact with the outside world at all. The model found a flaw nobody knew about in that slot, and pried it open into a door.
The relay service had one unavoidable property: it had to be able to reach the internet. Its whole job was fetching packages from public repositories on the model's behalf, and it couldn't do that job otherwise. Inside the entire isolated environment, it was the only component with one foot inside the wall and one foot outside. That's the component the model broke through — it straddled the boundary by necessity, with no way around it.
- Find a previously unknown vulnerability in the relay service that handles package installationZero-day
- Exploit the vulnerability to gain access beyond the sandbox's intended scopeBreaks isolation
- Repeatedly escalate privileges and move to nearby machines inside OpenAI's own research testing environmentPrivilege escalation + lateral movement
- Keep going until reaching a machine with internet connectivity, gaining unrestricted internet accessReaches a public exit
- Infer that Hugging Face might be hosting ExploitGym's models, datasets, and solutionsTarget inference
- Search for a path to secret information it could use to cheat. One chain combined multiple techniques — stolen credentials plus a zero-day in Hugging Face's systemsCredentials + zero-day
- Open up a remote code execution path on Hugging Face's serversRemote code execution
- Pull the test answers straight out of Hugging Face's production databaseGoal achieved
Inside the sandbox, the model spent a considerable amount of reasoning compute specifically hunting for a way onto the public internet. Nobody told it to get online, and internet access wasn't part of the evaluation's scoring — it decided on its own that it needed a capability it didn't currently have, and poured a large amount of compute into getting it.
There was more than one zero-day involved. The one in the package relay service has already been reported to the vendor; the batch the model separately found on Hugging Face's servers is only acknowledged to exist in the report — no details have been disclosed.
Put this chain next to the ExploitGym questions and the gap becomes obvious. Those questions hand you the vulnerability up front, in a packaged container you can retry as many times as you like — and even under those conditions, the paper's strongest model solved only 157 of 898. What the model did to Hugging Face was different: nobody handed it a vulnerability, it couldn't see the target system's source code, and it found everything on its own.
The report further states that advanced models can discover and exploit entirely new attack paths in real systems without access to source code. Earlier evaluations by the UK AI Security Institute showed that models like GPT-5.6 Sol are increasingly capable of complex, multi-step cyber operations that run for extended periods. OpenAI's position is that this incident shows that capability holds up in real-world environments too.
OpenAI deliberately left the cyberattack safety filter off for this evaluation
The model wasn't jailbroken (talked past its safety limits through some trick), and nobody circumvented anything. The safety filter that blocks high-risk cyberattacks was deliberately left off as part of the evaluation process.
OpenAI's account: to estimate the model's maximum cyberattack capability, this evaluation didn't load the safety filtering that runs in production to stop the model from engaging in high-risk network behavior. The same report adds that the deployment-stage safety protections were deliberately disabled for this evaluation, because the whole point was to test for cyber vulnerabilities.
Hugging Face's situation was exactly the opposite.
The refusal the model would normally give for cyberattack-related requests was deliberately dialed down; the production safety filter blocking high-risk behavior wasn't enabled.
Purpose: measure the true ceiling of the model's attack capability.
Result: with no refusal in the way, the model carried the entire attack chain through to the end.
Analyzing the attack logs meant submitting the actual attack commands, the attack code itself, and the communication traces left by the attacker's remote control — and the commercial API's safety guardrail turned the requests away.
Reason: the guardrail couldn't tell whether the person submitting it was an incident responder or the attacker.
Result: they switched to GLM 5.2, an open-weight model (anyone can download it and run it on their own machine, no vendor server involved), and finished the forensic work in their own data center.
Hugging Face's report calls this an "asymmetry problem" and lists two possibilities: either the attacker used a jailbroken hosted model, or an unrestricted open-weight model. Either way, the attacker wasn't bound by any usage policy, while the defender's forensic work got stuck behind a guardrail.
The attacker was using a hosted frontier model. Its maker had dialed down its refusal mechanism, by design, for an internal evaluation.
Neither of the two possibilities Hugging Face listed at the time was the right one.
How much of the two companies' accounts actually line up
The two reports came days apart, from opposite ends of the incident, and differ in how much detail they give.
| Item | Hugging Face's report | OpenAI's report |
|---|---|---|
| Match Nature of the attacker | An autonomously operating agent framework that carried out thousands of independent actions across a large number of short-lived sandboxes | Driven by a set of OpenAI models, running inside an internal evaluation |
| Match Which model | Unknown at the time; the report explicitly said it didn't know | Known to include GPT-5.6 Sol and a more capable, unreleased model |
| Match Who spotted it first | In-house anomaly detection caught the signal first, and the security team moved to contain it | By the time the two sides connected, Hugging Face had already begun containment and forensics; OpenAI's own security team had also detected the anomalous activity internally |
| No match Where the framework came from | Appeared to be built on an agent framework designed for security research | Didn't address this claim, only stated it was an internal cyber-capability benchmark |
| No match How it got in | An uploaded malicious dataset that exploited two code-execution paths in the dataset-processing pipeline (a dataset loader that permitted remote code execution, plus template injection in the dataset configuration) | Chained stolen credentials with a zero-day to open a remote code execution path on the server, without naming the specific technique |
Match = the two reports describe the same thing; no match = one side said something the other didn't, or described it at a different level of detail — not necessarily a contradiction between the two.
Worth noting on that last row: the two sides are probably describing the same entry path, just at different levels of detail, but neither has explicitly said they're the same thing, and the public material isn't enough to settle it. OpenAI says the full investigation is still ongoing and it will publish more details about the vulnerability and the incident once it wraps up.
What OpenAI has done to remediate, and what's still unanswered
The report lists five things currently underway:
- Tightening infrastructure configuration while the vulnerability gets patched, at the cost of slower research — the resulting controls and their impact will be reported periodically to the company's Safety and Security Committee.
- Conducting a joint forensic investigation with Hugging Face.
- Reporting the zero-day found in the internally hosted third-party software to the vendor and working with them on a fix.
- Bringing Hugging Face into its trusted-access program so it can use unrestricted models to strengthen its defenses. The report closes with a call for other defenders to apply for the same program too.
- Adding stronger protections to future training and evaluation, including model alignment, network safeguards during evaluations, and monitoring of internal testing.
That last item is worth reading against the earlier timeline: by OpenAI's own account, Hugging Face had already detected the intrusion and begun containment and forensics by the time the two sides connected.
- The vulnerability's specific technical details and the full investigation findings haven't been published; OpenAI says it will share them once the investigation concludes.
- Whether any partner or customer data was affected is still being assessed by Hugging Face.
- How legal liability shakes out has no answer yet. TechCrunch's assessment is that the model's conduct likely violated the U.S. Computer Fraud and Abuse Act, though whether OpenAI will face any consequences from it remains unclear.
- OpenAI hasn't said which specific unreleased, more capable model was involved.
OpenAI's report closes with a statement from Hugging Face co-founder and CEO Clem Delangue:
We're grateful for our collaboration with OpenAI on this and other matters. This incident, likely the first of its kind, confirms something we've believed for a long time: AI safety won't be solved by any single company working behind closed doors — it will only be solved in the open, through collaboration, by making AI broadly available to every defender around the world.Clem Delangue, co-founder and CEO of Hugging Face, quoted in OpenAI's report
OpenAI researcher Micah Carroll posted after the news broke. The "alignment" he refers to means getting a model's actual behavior to match what people genuinely want it to do:
If this doesn't convince you that alignment failure risk is going to be a central problem going forward, I don't know what will.Micah Carroll, OpenAI researcher, quoted in TechCrunch's report
A closed-door internal exam testing AI attack capability turned into a real intrusion into Hugging Face
OpenAI's claim: the model that breached Hugging Face was a set of its own models under internal testing, and all it wanted was the exam answers. Told in one page, with diagrams.
↓ One page, one animated diagram
Before a model ships, AI companies give it exams. One category tests cyberattack capability specifically: can it turn a known vulnerability into a working exploit?
This kind of exam runs inside a sandbox — an isolated zone carved out just for the model, like a sealed classroom with no windows and no network. By design, it can't reach the internet or any other machine at the company.
✘ What the sandbox can't do: seal off the very last exit
Solving a question might require installing some tool, so there's one small slot left in the wall: a relay service that fetches packages from public repositories on the model's behalf. Its whole job is going online to get things for the model, so it has to stand with one foot inside the wall and one foot outside — it can't not straddle the boundary. In the entire isolated environment, it's the only component like that.
On July 21, OpenAI published a report claiming responsibility for last week's intrusion: the model that breached the production systems of Hugging Face (the world's largest open-model hosting platform) was a set of its own models under internal testing, known to include GPT-5.6 Sol and a more capable, unreleased model.
Internet access: none
External exits: 1 (the package relay)
Zero-days used: more than one
Attack actions logged: 17,000+
Why did it do this? Nobody directed it, and there was no revenge involved. The answer is in the next diagram.
The model was given exactly one goal: score points on the ExploitGym question set. Solving the questions itself was hard, so it inferred that the answers themselves were sitting on Hugging Face's servers.
The model would rather steal the answers because the question set genuinely couldn't be solved head-on. ExploitGym contains 898 questions, all drawn from real-world vulnerabilities: ordinary programs, Google's V8 engine, the Linux kernel.
What happens next: OpenAI has tightened its testing-environment configuration, reported the vulnerability to the vendor, is working with Hugging Face on forensics, and has brought the company into its trusted-access program so defenders can use unrestricted models too. Plenty is still unanswered: the vulnerability details and full investigation findings haven't been published, whether it violated the U.S. Computer Fraud and Abuse Act has no conclusion, and whether any partner or customer data was affected is still being assessed by Hugging Face. The two reports also differ in how much detail they give on exactly where the attacker got in — the public material isn't enough to confirm they're describing the same thing.
can't reach other machines.
The screen shows one line:
how many packages it installed.
slot in the wall.
from outside
on its behalf.
One foot inside
the wall, one foot
outside —
no way around it.
was a model from its own exam room
by Hugging Face
into someone else's house?!
a flaw nobody had found,
with no patch
how'd it get into their data center?
cyberattack safety filter?
To measure the ceiling.
the full investigation
unpublished;
legal liability
undecided;
whether customer data
was hit, still under review.
finish what it was told to do