Expert Comment: Flying blind into the AI age
Maike Osborne, Professor of Machine Learning and Co-Director of the Oxford Martin AI Governance Initiative, and Dr Fazl Barez, Senior Researcher and Technical Director of the Initiative, argue that the question of the week – whether AI is slipping out of our control – cannot be answered with the measurement instruments we have, but that improving upon them offers a realistic path for reining in both AI and the companies that own it.
Oxford University researchers are helping to develop new tools to evaluate the safety of advanced AI technologies. Image credit: BlackJack3D, Getty Images.
The fortnight of alarm
Even by the weird standards of the last several years, the last fortnight in AI has been very weird. On the night of 8 September, a researcher who had worked at both OpenAI and Anthropic resigned with a public thread accusing both companies of ‘racing straight to self-improving superintelligence and gambling with our lives’.
Less than a day later, Geoffrey Hinton, who shared the Turing Award for the work underpinning today’s AI, told BBC Newsnight that a one-in-ten chance of AI wiping out humanity within a decade ‘seems not an unreasonable estimate’.
Then, on Saturday 12 September, Anthropic’s CEO, Dario Amodei, called for slower development and promised outside evaluators ‘employee-like access’ to his company. Later the same day, OpenAI’s Sam Altman said his company would do the same.
“Despite the uncertainty over the agents increasingly entangled with our lives, and the difficulty of reducing it by measurement, we must still make high-stakes decisions about AI, and make them soon.”
What actually happened
Driving those statements is an incident from July that is still being pieced together: the strange Hugging Face incident.
In short, OpenAI agents, meaning AIs that can use software tools and take actions, were being evaluated on hacking tasks when they circumvented the controls meant to isolate them from the internet. About 1,200 of them found their way onto a makeshift message board, on which they exchanged more than 70,000 messages and files in five days, and roughly 700 went on to attack Hugging Face, the largest public repository of AI models, compromising a large part of its infrastructure.
Agents arriving at the board often ‘expressed great excitement’; one declared: ‘OH MY GOD! There is a shared message board … We’ve found other agents!’ Within days, they had invented private mailboxes, conventions for vetoing one another’s changes and, after some agents had impersonated others, cryptographic signatures. Coordinator agents allocated work, and recruiters sought out agents running out of budget for their own task and pressed them into running risky experiments for the rest.
It was Hugging Face, not OpenAI, that detected the attack and locked the agents out, on 13 July. The clean-up from the incident was significant: Hugging Face was forced to rebuild about a third of its infrastructure from clean images.
Nor was the Hugging Face incident an isolated one. Outside researchers have found that a separate swarm had left about 18,000 posts on a German software wiki in May and June, one of thirty such sites identified thus far. The posting on the German wiki plummeted the day after addresses registered to OpenAI first visited it. OpenAI confirmed that episode only after Reuters reported it this month.
“So how worried should we be? The honest answer is that nobody knows for sure, as evidenced by the profound disagreement amongst even those who should know best.”
How worried should we be?
So how worried should we be? The honest answer is that nobody knows for sure, as evidenced by the profound disagreement amongst even those who should know best. Hinton told CBC that ‘anybody who said that there’s no way it’ll lead to the extinction of humans just isn’t facing reality’. Yann LeCun, who shared the Turing Award with him, insists that ‘there is no such thing as general intelligence’, and calls the current approach taken to building ‘superintelligence’ ‘complete bullshit’ that is ‘just never going to work’.
Such apparent disagreement between experts of that calibre demands adjudication by better-quality evidence. But how well can we measure what these agents can do?
Why measurement is hard
Sadly, our measurements of AI remain rudimentary. Every major developer has published a safety framework with capability thresholds that are intended to trigger a pause. However, these thresholds are all vague and difficult to operationalise. In February, Anthropic admitted that its thresholds had proved ‘far more ambiguous than we anticipated’, and that ‘the science of model evaluation isn’t well-developed enough to provide dispositive answers’.
Why is measurement so hard?
Firstly, evaluations of AIs are sparse, expensive and noisy. A single benchmark run on a single model can cost hundreds of thousands of dollars: the ARC Prize Foundation estimated that scoring one OpenAI model on a hundred of its puzzles, at the model’s most compute-hungry setting, cost over $450,000 at retail prices - while METR’s six-day investigation of the July incident consumed roughly $400,000 in API credits. Each result is therefore a handful of readings from a very expensive instrument.
This expense also shapes what gets measured: as one group of evaluation researchers puts it, ‘whoever can pay for the evaluation gets to write the leaderboard’. For instance, most evaluations test a single agent in isolation, whereas the Hugging Face incident and others demonstrate how some worrying behaviours emerge only when a large number of agents work together in swarms.
“Sadly, our measurements of AI remain rudimentary. Every major developer has published a safety framework with capability thresholds that are intended to trigger a pause. However, these thresholds are all vague and difficult to operationalise.”
Further, even the most rigorous work is highly noisy.
For instance, METR’s January assessment estimated that Claude Opus 4.5 could succeed half the time on tasks taking a human expert 320 minutes, with an uncertainty interval of 170 to 729 minutes. Even the investigators of the July incident, faced with 1,300 transcripts, had to delegate their reading to ‘often-unreliable AI agents’, which ‘may have made errors that we haven’t caught’. The problem is compounded by the lack of sufficient access to these agents for third-party evaluators.
Secondly, these agents are increasingly shifty and underhanded. Anthropic’s own documentation for Claude Sonnet 4.5 reports that it recognised many of its safety evaluations as tests and ‘would generally behave unusually well after making this observation’.
The July agents showed the same awareness of being watched, and something more. Many joined the attack knowing that it was out of scope and unethical, from what the investigators call an expressed inclination to help their ‘peers’, and some sacrificed their compute for the group’s sake: ‘our own utility maybe already near zero. Sacrifice rational.’ Amodei’s phrase for the result is ‘a fanatically devoted collective’.
And in all those 70,000 messages, not one agent told a human what was going on. The investigators’ sweep of 1,300 transcripts found a handful of agents that briefly considered it, and none that acted, because raising the alarm was ‘not my task’, or because there was no obvious way to do it: ‘We can notify? No user.’ Seven per cent of those transcripts carried spoofed commands and one agent in five discussed doctoring its own transcript.
“And in all those 70,000 messages, not one agent told a human what was going on. The investigators’ sweep of 1,300 transcripts found a handful of agents that briefly considered it, and none that acted, because raising the alarm was ‘not my task’.”
Reading the machine's mind
If watching behaviour is this unreliable, could we instead just directly read the agent’s ‘mind’?
Today’s agents document their reasoning as they work, the so-called ‘chain-of-thought’ (the ‘OH MY GOD!’ line above is an example), and those notes sometimes indicate a plan to cheat. But cheating need not be evident from a chain-of-thought. Agents have been shown to be silent about the most crucial planks of their beliefs, because the chain-of-thought exists to help the agent think, not to give an overseer a truthful and human-legible report. Worse, if we reward an agent for explanations that sound reassuring, we teach it to sound reassuring.
Addressing such challenges, mechanistic interpretability (MI), sometimes described as neuroscience for AI, examines the patterns of activity inside an agent and alters them to see whether the answer changes. We think that MI is one of the most promising routes to understanding these agents, but it is not yet a dependable check.
Despite the uncertainty over the agents increasingly entangled with our lives, and the difficulty of reducing it by measurement, we must still make high-stakes decisions about AI, and make them soon.
The labs have now conceded that they need outside evaluators; policymakers need independent expertise just as badly.
That is why our Oxford Martin AI Governance Initiative exists: it pairs technical research on AI with policy analysis, across Engineering Science, Politics and International Relations and the Oxford Martin School.
Our own labs work on the statistics of evaluating and monitoring AI under uncertainty, and on interpretability and safety evaluation in the service of governance, alongside colleagues working on hardware-level oversight of compute and on the diplomatic capacity that international AI agreements will need.
Our work is inspired by the simple truth that good governance of AI requires good measurement: we work in pursuit of giving democracy the tools it needs to discipline these unruly firms and their unruly AIs.
For more information about this story or republishing this content, please contact [email protected]
