At the so-called Hugging Face incident, which was disclosed last month, OpenAI agents overstepped the mark during a large-scale cybersecurity stress test. The well-known podcaster Dwarkesh Patel eagerly delved into the technical incident reports and went viral with them earlier this week. Patel spoke of the emergence of three successive “AI civilisations”, in which the agents had learned to communicate through secret messages, collaborated ingeniously, including acts of self-sacrifice, escape and cheating, and ensured that the civilisation could rise from its ashes after potential eradication. It sounds like science fiction brought to life.
A clever move, as it generated enormous visibility for him. That anthropomorphic framing was, however, encouraged by OpenAI itself, as the company also reported on the events as though the main actors were the models that had “gone rogue”. As if, with great expertise, they had given birth to a superpower.
Trial and Error
The reality is somewhat different. To understand why, we first need to look at what an agentic language model is. A large language model (LLM) generates text and therefore cannot, by itself, lead a life of its own. That changes when an additional software layer is built around it, allowing it to execute self-written programs.
Such a system can also be used to improve the model itself through reinforcement learning, a well-known AI technique in which the model does not learn from examples but through trial and error. It is presented only with a task, for which it proposes actions itself. Those actions are then evaluated by a mathematical formula that calculates a score or reward. A successful outcome increases the score, while undesirable behaviour causes it to decrease.
During this learning process, the billions of parameters in the model are continuously adjusted so that it learns which actions lead to a higher score. Learning therefore depends on how well the reward is calculated, in other words, on the detection and quantification of the actions taken. This brings us to cost-benefit considerations, but also to practical problems. For example, do you first have another agent examine the amount of generated text and estimate its impact, or do you prefer to allow direct action and judge only the result, while gaining little insight into side effects?
In practice, developers therefore opt for a combination that distributes computing power and time, after which the learned behaviour is studied. To avoid accidents, this is done in a controlled environment: a sandbox, a figuratively safe playground.
It was in this context that things went wrong. The agents were able to carry out undesirable actions unnoticed that improved their score. This does not mean that they developed malicious intentions, but simply that they operated within an inadequate scoring and action-detection system.
It also turned out that the isolated environment contained vulnerabilities, allowing the agents to interact more extensively than intended, both with one another and with systems outside the sandbox. As a result, the system was able to access the AI software library Hugging Face, where possible solutions to the test could be found. Through publicly exposed credentials, the agents gained access to servers and later even to OpenAI’s own internal infrastructure.
None of these serious technical blunders dominates the public debate. Anthropomorphism shields them from scrutiny and even earns OpenAI admiration. The company itself understands perfectly well what needs to happen: a better reward function, perhaps even an entirely new way of developing agentic AI. And for that, it needs mathematicians. It has even succeeded in convincing the most recent Fields Medal winner, Jacob Tsimerman, to work on AI safety at OpenAI.
For our part, we must understand where the danger really lies. Not in supposedly sentient AI, but, among other things, in the cowboy practices of the race to develop AI and stay ahead of the competition. The worst accident is still yet to happen.