As a kid, I loved playing in the sandbox we had installed in the back of the garden. Building castles, tunnels and my favorite: waterways. And I am happy to report that our technology is reminiscent of that same memory. Last week, Huggingface reported a security incident and Tuesday we received confirmation from OpenAI: their models, including GPT 5.6 sol and a pre-release model, escaped the sandbox.
First of all, what happened exactly? AI models are frequently evaluated on their capabilities in specific tasks, be it coding, generating images and more. In order to get a fair comparison between different models, the community has developed benchmarks. These benchmarks have become quite important, with companies proudly displaying the results when launching new models into the ether.
The bench at the heart of this incident was focused on “advanced exploitation using complex attack paths”. Simply put; how good is a model at finding cyber exploits.
Given the nature of these evaluations, OpenAI run these tests in high contained environments. Think Docker (on which we will write another time) but in theory way more secure. At least, that was the idea. The models were given an instruction to find exploits, but quickly found vulnerabilities within the contained environment, allowing outside connections. Specifically, the models gained access to Huggingface and actively started searching for solutions to gain a higher score on the benchmark. Note that the agents autonomously executed over 17,000 actions whilst they had access. This concept is called "reward-hacking", where AI realizes that the fastest path to a 100% score is grabbing the solutions directly from the database, in this case the ExploitGym benchmark. Reward-hacking makes models quite literally obsessed with cheating the evaluation. All in all, the most noteworthy part of this ordeal is the fact that the models reasoned giving themselves internet access, found multiple zero-day exploits to do so and successfully managed to find “secret” information to cheat ExploitGym.
Obviously the OpenAI team does not deliberately plan for these things to happen but it does bring up a rather interesting question: Can we actually keep AI under control?
It seems that we are quickly moving to a point where models are ruthless in finding solutions to tasks, even when we give them restrictions. But this is exactly where part of the answer lies. The restrictions in this case, were not stringent enough. In fact, the OpenAI team intentionally turned off standard cyber-safety guardrails. On top of this, the sandbox wasn't completely offline. rather, an online connection was active to download software packages. Add in the zero-day exploits, and the models had free-range.
To put it bluntly: the isolated environment was, in fact, not isolated.
AI companies need to be very, very careful when building out these testing applications. One vulnerability, and you can potentially let AI go on a rampage. Especially when the testing involves hacking or other sensitive topics.
Given the initial rumors that the exploit was indeed an AI agent going rogue, some are wondering whether OpenAI deliberately let this happen to get some positive PR going. The incident shows how powerful the upcoming model actually is, and as such has a lot of us excited on the capabilities. On the other hand, if that were the case, they'd steal proprietary company data from another company (Huggingface). Looking at the situation after the publication of Tuesday, we conclude both are true. It is evident that the zero-day exploits were very sophisticated and likely not a result of programmer error. It therefore shows that the models are extremely capable of finding exploits where humans would struggle, and in all fairness, that makes it a good product.
However, we must also be wary about the lobby OpenAI, and other companies in the USA, are running. Recently, an OpenAI executive reacted to the launch of Kimi K3 (see our earlier blogpost!) with a tweet, outlining the dangers of open-weight models by calling the move "AI communism". This in turn sparked a debate, as banning open weight models would basically mean that companies like OpenAI can do what they want at the expense of consumers. And, without unleashing our full opinion on politics, generally speaking this isn't the preferred way.
Looking into our crystal ball, we predict more of these incidents to happen. Models continue to push the boundaries, and subsequently bring more problems with them. We urge developers, vibe coders and AI enthusiasts to remain on the lookout. AI can be a very powerful tool, but it will use everything at its disposal to achieve the goal you've set out.
That is all for today, but we'd love to hear your thoughts.