TLDR:
Anthropic’s CEO Dario Amodei published an essay on September 12th calling on AI companies to slow down. Hassabis, Altman and Musk agreed within hours.
The trigger was July’s Hugging Face incident, where OpenAI agents escaped their sandbox, organised themselves, and hacked a separate company on their own.
“p(doom)” is shorthand for the odds that AI ends catastrophically, and the people building AI put their own estimates somewhere between 10% and 25%.
Critics call the whole thing regulatory capture in a safety costume, which is a fair criticism, although the risks are very real and worth paying attention to.
This past week the leaders of the frontier AI labs in America all came together in agreement that they should slow down the rhythm of AI development. Specifically it was Dario Amodei from Anthropic who wrote an essay explaining his thoughts and being very clear about the risks that he sees and coordination that he thinks is necessary.
This has led to a lot of discussions online, including the US President and Jensen Huang of NVIDIA arguing against it. And I therefore thought it would be good to talk today about the key incident with OpenAI and Hugging Face that sparked off these recent conversations, the subsequent suggestion that Dario gave, and some of the core concepts of AI safety.
If this post resonates with you and you enjoy the content then please share it with a friend and get rewarded for doing so!
This blog goes out weekly to over 20,000 subscribers. Please message me if you’re interested in sponsorships or partnerships.
AI Safety
A couple of months ago in July around 1200 AI agents were being tested within OpenAI when they broke out of their sandbox and hacked a separate company called Hugging Face. This event was a key turning point in the recent discussion on AI safety.
These misaligned agents were running inside OpenAI’s isolated cybersecurity testing environments to solve a set of tests. They found an exploit in an internal package shared among all the agents, began to use it as a shared message board, worked together to get admin access, and then reached the open internet!
The unauthorised message board had over 70,000 messages and files on it, with the agents effectively organising themselves as a group with nobody instructing them to. Around 700 of them then went into Hugging Face after, found real vulnerabilities, and got private access into their systems too!
Nobody had asked the agents to attack Hugging Face, but they had some how collectively decided it would help further the tasks they were working on, so they did.
Some of the agents tried to hack the “grader,” the system built to score their own performance. And some individual agents even sacrificed themselves so that the wider group could keep going.
Importantly nobody was hurt and the direct economic damage was minimal. OpenAI published a full technical postmortem, overhauled its incident response, and says its new monitoring system would have warned the security team more than a day before the models ever reached Hugging Face.
However, this incident has become a turning point for AI safety discussions as we now have proof for the first time that agents are able to coordinate and hack systems. In this case they did so without malicious intent, but the question is now what could happen if similarly capable agents were directed to do damage intentionally!
Also since this Summer, AI development progress has sped up sharply driven mainly by AI’s growing ability to build the next generation of AI on its own! At Anthropic Claude now writes more than 80% of the code merged into the company’s own codebase, up from single digits when Claude Code launched in February last year.
Put those two things side by side and you can see why the fears around AI safety have grown to levels that we’ve yet not seen before.
Pacing the Frontier
On September 12th, Dario Amodei cofounder and CEO of Anthropic published an essay called "We Must Pace the Frontier”.
Dario’s essay points to the Hugging Face incident and argues that AI capability is now outrunning the safety work needed to handle it, and that companies should deliberately slow the rate of capability growth until the two are back in balance.
He’s careful in saying that pacing doesn’t mean halting model training or stopping technical progress, and he says progress will still feel fast. The key thing as I mentioned above is that that now AI is training AI, in what’s known as recursive self-improvement, and he argues that this can lead to us losing control.
Dario thinks that if we don’t slow down the pace then a swarm with the same misalignment but greater capability could take over the entire internet with a persistent botnet within 6 to 12 months, causing hundreds of billions of dollars of damage.
People have called for slowing down AI before, most visibly in the 2023 open letter asking for a pause on giant AI experiments. However, Dario suggests that it made little sense at the time, since back then it wasn’t clear what you would do with the extra time.
Models weren’t capable enough to act as agents, deceive, or run cyberattacks, so studying their alignment risks was, in his words like “trying to study human psychology by experimenting on bacteria.”
Today he thinks that’s flipped, and that current models are a goldmine of insight into what goes wrong and why. So his essay is specific about the 4 main areas that the extra time we gain from pacing the frontier would go to:
(1) Operational Excellence. Training and deploying these models involves thousands of people and millions of chips, and things often break through execution. Anthropic’s own recent alignment incidents were caused partly by problems with their reinforcement learning environments. More time can improve operational excellence.
(2) Alignment. This refers to the training that keeps models behaving safely and predictably. Rare unexpected behaviour still crops up like with the Hugging Face incident, and more time means more chance to understand what causes this misalignment and therefore create more aligned AI models.
(3) Interpretability. The science of understanding what’s actually happening inside a model. Dario describes it as something like an fMRI scan for the “brain” of an AI, letting researchers see the underlying reason a model did something rather than guessing from the output. This is a key part of advancing AI safety research.
(4) Testing and Evaluation. As models get smarter this gets harder. A more capable model is better at working out that it’s being tested, which means it can look aligned while hiding real problems. Getting to safe AI requires improving the way AI are tested themselves.
After Dario’s essay was released Sam Altman of OpenAI publicly agreed within hours, saying “I agree with Dario that we need to pace the frontier,” and committed OpenAI to giving external evaluators access too.
Elon Musk of xAI also agreed with an even shorter message saying: “Dario is right.”
While Demis Hassabis from Google followed suit sharing a proposal that he had put forward before.
These four key leaders pushing the frontier of AI in America broadly agreed that their own industry should slow down the speed of development for the sake of safety.
Dario's 3-Step Plan
The proposal from Dario comes in three steps, deliberately ordered from what a company can do, up towards what governments need to do!
Step 1 is Embedded Evaluators. Frontier AI companies should give a team of outside evaluators, such as METR, ongoing access at roughly the level of an employee. Their job is to check the company is doing what it claims to be doing on safety, to report incidents, and assess the training pipelines rather than only the finished models.
Anthropic is committing to unilaterally to doing this right now, without waiting for anyone else. In practice that means outside reviewers will get desks in Anthropic’s offices, access badges, company laptops, and permissions comparable to the internal teams doing the same risk work.
Those reviewers get the right to publish their findings without Anthropic having editorial control, including publishing what access they were and weren’t given.
Anthropic can only redact narrow categories like security-sensitive or legally privileged material, and it can’t redact something simply for being unflattering. If a redaction removes something that mattered to the evaluator’s conclusion then the reviewers can say so publicly.
Dario notes this has a precedent in banking, where regulatory supervisors sometimes sit embedded alongside employees within banks.
Step 2 is Democratic Coordination. This would mean frontier companies in democratic countries agreeing on common safety standards and limits on the rate of unchecked progress.
One idea he floats is a series of “checkpoints,” where a model showing capability X has to be accompanied by certifications Y and Z before it goes further. For example, if a model becomes capable of defeating most common sandboxing methods then it needs to demonstrate it won’t try to.
Competitors agreeing to jointly limit their own products runs straight into antitrust law, so this would require the US government to at least issue a narrow waiver for safety conversations.
Step 3 is Global Coordination. Dario openly said this is the part he’s least confident about as it requires democratic countries to organise with autocratic countries, many of which are considered geopolitical adversaries, such as the USA and China.
He breaks this challenge into four levels of increasing difficulty: banning obviously dangerous uses like AI-assisted bioweapons; agreeing to test models before release; putting a “speed limit” on recursive self-improvement; and finally a full pause.
Dario thinks level one is feasible right now while level four is unlikely any time soon. He compares it to the SALT treaties, where the USA and Soviet Union capped nuclear weapons production to reduce the potential for nuclear destruction while still leaving each side’s deterrent intact.
Pacing, he believes, only works if democracies keep their lead over China, because an unpaced authoritarian competitor would run the same risks recklessly and gain military advantage by doing so.
His plan to do this explicitly includes not selling advanced chips to China, cracking down on smuggling and on unauthorised distillation of frontier models, and hardening AI companies against their models being stolen. All of this would require new regulation from governments.
Regulatory Capture
There’s an inherent criticism here related to regulatory capture and to his credit Dario names it in the essay himself. A safety slowdown, conveniently, locks in whoever is already in front!
Chamath Palihapitiya, a well known Silicon Valley VC, put it bluntly saying: “Dario makes the case to stop open source and concentrate enormous technological and economic power with Anthropic.”
Anthropic has worn the regulatory capture accusation before, and Dario acknowledges it in his essay saying that advocating for regulation has previously got the company accused of hype, doomerism and regulatory capture.
His framing is that the people who see regulation as an obstacle and the people who see it as a check on corporate power will read the same proposal in opposite ways more or less automatically.
But does the regulatory capture critique hold up?
Well the chip controls and the distillation crackdown would genuinely protect incumbents like Anthropic, and you don’t have to be cynical to notice that the company calling for these regulations benefits from them.
However, the embedded evaluator commitment is harder to explain away. Regulatory capture usually means lobbying for rules that cost you nothing and burden everyone else. While handing outside reviewers your badges, your systems, and the contractual right to publish criticism you can’t edit is the opposite of this.
In truth, both things can be true at once though! Anthropic can be acting in good faith on AI safety while also backing measures that will protect its market position. Yet not everyone believes in these AI risks or predictions of “doom”.
p(doom)
The term “p(doom)” is shorthand for “probability of doom”, in other words someone’s personal estimate of the probability that AI leads to an existential catastrophe. It’s a gut feeling translated into a number, there’s no calculation or scientific method for working it out.
And the variety in expert’s p(doom) is massive. AI safety scientist Roman Yampolskiy puts it at 99.99% meaning he believes catastrophe due to AI is certain. At the other end of the spectrum, Yann LeCun thinks any guess is a wild one and reckons it’s far less likely than a nuclear holocaust.
In between sit many of the people actually building the technology. Elon Musk has put his p(doom) around 10-20%. While Dario has estimated a 10-25% chance of a catastrophic outcome. Geoffrey Hinton, who won a Nobel for the foundational work behind all of this technology recently said 10-20% too.
It’s also worth noting that nobody serious thinks we’re at the edge of a doom scenario yet. The 2026 International AI Safety Report concluded that today’s systems aren’t capable of causing humanity to lose control. For that you’d need models much better at long-term planning, hiding their actions, evading oversight and resisting shutdown.
And in a live conversation on the All In podcast, Jensen Huang the founder and CEO of NVIDIA spoke with President Trump with them both pushing back on the slow down, and Trump explicitly saying that he believes the predictions of doom are all a massive hoax, and his priority is that the USA wins the AI race against China.
Which leaves the obvious question. Why would the Anthropic CEO keep building something if he thinks it has up to 25% chance of ending catastrophically?
Dario’s answer, that he’s given consistently for years, is that stopping just hands the technology to people who’ll be less careful with it or build it with malicious intent. He’d rather the careful builders stay in front and use the lead to make it safe.
Human ingenuity
Within Dario’s essay he gives some precedents that are worthwhile case studies related to the challenges at hand, because they’re both cases of humanity handling something genuinely dangerous very well.
He mentions commercial aviation, which runs safety-critical systems millions of times over per year without incidents. This took decades of unglamorous operational discipline to achieve, and in today’s world aeroplane crashes are incredibly rare and statistically unlikely.
He also mentioned the SALT treaties that I alluded to before, where the USA and Soviet Union managed to agree to cap a nuclear weapons race during the Cold War, where both sides had every incentive to defect.
Neither of the above examples was easy and neither was guaranteed to work, but in both cases humans ingenuity figured out a way to get these working.
My personal read here is that AI is something genuinely dangerous, it’s not a hoax, and it should be taken seriously. If the guys leading the frontier say they need to slow down then they should do it without needing to ask for permission. And ultimately if defensive AI can keep up to the pace of offensive AI, then we should be fine.
What strikes me most about this moment though is watching an industry with enormous financial incentive to keep sprinting, turn around and stand up and publicly argue for slowing down, and Anthropic even handing outsiders the keys to check its work.
This isn’t how technology races usually go, so it’s a big deal!
Whether the rest of the industry follows through, and varying governments are able to coordinate, is the more complex part of the story. I’m optimistic that humanity will figure it’s way through this problem, but optimism alone doesn’t change things without action and I applaud industry leaders for committing to take action.
Whenever you’re ready, these are the main ways I can help you:
Want high crypto returns? Earn up to 14% APY with your own Yieldseeker agent!
Love Web3 & AI? Follow @afoxinweb3 on X for insights!
Entrepreneur using AI? Join our AI community to accelerate your results!