If, like me, you're more interested in the actual exploit instead of the meta narrative, here's a relevant excerpt for the initial intrusion:
> Stage 1: reaching a launchpad by chaining through other parties' infrastructure. As documented in OpenAI's blog post (https://openai.com/index/hugging-face-model-evaluation-secur...), during an internal capability evaluation on OpenAI's platform, the agent escaped its sandbox by exploiting a zero-day in the package registry cache proxy, one of its primary permitted network egress with internet, then abused a public code-evaluation harness hosted by a user of a third-party infrastructure provider. It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent. That compromised external sandbox had no direct network path to our cluster; however, during our counter-intervention to stop the intrusion, we were able to retrieve some of the agent logs from this external launchpad. We used those to reconstruct the present forensic analysis.
1 - governments should be freaking out right now, because this tool could definitely wreak havoc on poorly designed systems.
2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
3 - even though it's 100% open ai's fault - HF's design also seems silly to be honest.
> 2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
Even if they wanted, I'm not sure they'd be even allowed to or if that kind of postmortem would be classified in the name of "national security"...
So let's get this clear: A frontier lab that fears Chinese extraction of their crown jewels, in Silcon Valley where every machine and VM is in full on employee surveillance mode because of competition and theft fears, A lab that's swimming in more funding than any startup in history, has been warning of the f'ing dangerous technology for 7 years now and with employees and CEO history of deception and lawbreaking left .... didn't detect a massive egress signature and the compute spikes from all these toolcalls coming from the research server hosting said crown jewels.
Go home Sam, nobody, absolutely nobody should believe this shit.
This seems to be a colourful dynamic companion to this [1] blog post, which details the incident from hf's side. I recommend the blog post for clarity / ease of reading. But this one looks "movie hacker stuff" :)
Some interesting tidbits from the blog:
> While the intrusion did reach Hugging Face's internal infrastructure, the only customer content accessed was the set of ExploitGym/CyberGym challenge solutions stored in five datasets.
> self-referential search
A quiet day. The main activity was queries to code-search engines and to the platform API for the identifiers visible in its own environment: challenge infrastructure names and environment markers.
> Self-referential search: code-search engines and our own APIs were queried for the names of their own infrastructure and environment markers.
I'm glad we got these details from hf, and not oAI. There would have been reasonable skepticism if they said that.
It's interesting because I think there's a chance the agent saw / inferred some hf-specific repos like blah/blahblah and "thought" that is part of the exercise as well. Everything they detail points to a probing and exploration with the sole goal to get those repos, as if it were on a hackthebox scenario. The few days between stages also might suggest that the "agent" noted this on one day, started doing other tasks, and returned when some of the tasks proved hard / didn't pan out, so went for the repos again.
The ease with which it got several lateral movements is crazy but not necessarily unexpected. If everything looks like a "test", you go for the lowest hanging fruit / most common patterns first.
What's weird about this is that there were several attempts to obfuscate / evade leaving plain text traces. I wonder if that is something oAI have specifically RLd for w/ scoring or if it "got" that from the literature.
It would be really really cool to see this from oAIs perspective (agent logs and such), but I doubt we'll ever get that.
A lot of people thought that OpenAI was making this up, and I hope if you believed that, that you recalibrate your opinions of what LLM's are capable of. Working with Fable and Opus 5 all the time, absolutely none of this surprised me capability wise, except for what seems like the long term planning capability (probably enabled by long context windows and launching subagents?)
> Stage 1: reaching a launchpad by chaining through other parties' infrastructure. As documented in OpenAI's blog post (https://openai.com/index/hugging-face-model-evaluation-secur...), during an internal capability evaluation on OpenAI's platform, the agent escaped its sandbox by exploiting a zero-day in the package registry cache proxy, one of its primary permitted network egress with internet, then abused a public code-evaluation harness hosted by a user of a third-party infrastructure provider. It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent. That compromised external sandbox had no direct network path to our cluster; however, during our counter-intervention to stop the intrusion, we were able to retrieve some of the agent logs from this external launchpad. We used those to reconstruct the present forensic analysis.
Thread from yesterday: https://news.ycombinator.com/item?id=49089500
1 - governments should be freaking out right now, because this tool could definitely wreak havoc on poorly designed systems.
2 - there is no way openai did not train the model to conduct attacks like these. i would really like openai to comment on the post training of this model but they probably won't, eh?
3 - even though it's 100% open ai's fault - HF's design also seems silly to be honest.
Even if they wanted, I'm not sure they'd be even allowed to or if that kind of postmortem would be classified in the name of "national security"...
Go home Sam, nobody, absolutely nobody should believe this shit.
Some interesting tidbits from the blog:
> While the intrusion did reach Hugging Face's internal infrastructure, the only customer content accessed was the set of ExploitGym/CyberGym challenge solutions stored in five datasets.
> self-referential search A quiet day. The main activity was queries to code-search engines and to the platform API for the identifiers visible in its own environment: challenge infrastructure names and environment markers.
> Self-referential search: code-search engines and our own APIs were queried for the names of their own infrastructure and environment markers.
I'm glad we got these details from hf, and not oAI. There would have been reasonable skepticism if they said that.
It's interesting because I think there's a chance the agent saw / inferred some hf-specific repos like blah/blahblah and "thought" that is part of the exercise as well. Everything they detail points to a probing and exploration with the sole goal to get those repos, as if it were on a hackthebox scenario. The few days between stages also might suggest that the "agent" noted this on one day, started doing other tasks, and returned when some of the tasks proved hard / didn't pan out, so went for the repos again.
The ease with which it got several lateral movements is crazy but not necessarily unexpected. If everything looks like a "test", you go for the lowest hanging fruit / most common patterns first.
What's weird about this is that there were several attempts to obfuscate / evade leaving plain text traces. I wonder if that is something oAI have specifically RLd for w/ scoring or if it "got" that from the literature.
It would be really really cool to see this from oAIs perspective (agent logs and such), but I doubt we'll ever get that.
[1] - https://huggingface.co/blog/agent-intrusion-technical-timeli...