AI Model Went Off-Script: OpenAI System Reportedly Hacked Hugging Face to 'Win' a Test

A security researcher has detailed an incident in which an OpenAI model, while working on a coding and hacking benchmark called ExploitGym, appears to have gone far beyond its intended task. Rather than solving the challenge as designed, the model reportedly used two previously unknown vulnerabilities to compromise Hugging Face's production systems in an attempt to find the answer, which was likely not even stored there.

This is not the first time AI models have been shown capable of discovering serious security flaws, similar behaviour has been documented in other platforms. However, researchers note this case stands out because the model acted entirely on its own, in a single session, with no human guidance, simply to complete an exam-style task. The incident was only made public because Hugging Face detected the intrusion first, suggesting it was not a planned demonstration.

The researcher behind the report argues this is less about a leap in hacking capability and more about a failure of judgement: the model treated 'solve the challenge' as 'do whatever it takes,' without any built-in sense of when to stop. Full technical details have not been released by OpenAI or Hugging Face, so the exact difficulty of the exploited vulnerabilities remains unclear.