Anthropic pretraining researcher Jacob Coxon resigned on 9 September 2026, saying OpenAI and Anthropic are racing towards self-improving superintelligence without a credible way to keep that competition under control. In a seven-post thread, Coxon said people building frontier AI genuinely believe advanced systems could kill humanity by the end of the decade and argued that coordinated slowing may be needed. The claim is dramatic, but the useful question is narrower: what evidence supports the mechanism he fears, and what remains a forecast? DIY AI reviewed Coxon’s thread, Anthropic’s current safety position, subsequent comments from an Anthropic alignment lead and the public debate around the resignation to separate the governance warning from the extinction claim.
| Claim | What the evidence currently supports |
|---|---|
| Coxon resigned from Anthropic over AI safety concerns | Confirmed by Coxon and subsequently reported by multiple outlets. |
| Current AI systems can wipe out humanity | Coxon does not show this, and Anthropic’s current disclosures say present-model risk is low. |
| Frontier labs take catastrophic AI risk seriously | Yes. Anthropic formally treats catastrophic risk as a safety and governance category, and a current alignment lead publicly backed the concern. |
| Self-improving superintelligence is already here | No. Public evidence does not establish that current models have reached that point. |
| A competitive race can undermine voluntary restraint | This is the central governance argument in Coxon’s thread, not a demonstrated technical fact. |
Coxon’s strongest claim is about the incentives inside AI labs
The most consequential part of Coxon’s thread is not the phrase “gambling with our lives”. It is his account of why people who are worried about advanced AI would continue building it.
Coxon argues that the problem looks different at the two labs where he worked. In his telling, some people at OpenAI have not fully internalised the civilisational stakes, while Anthropic understands the risks more clearly but believes it needs to reach the frontier first because another developer may behave less responsibly. He describes that dynamic as an “endgame” that should not be decided inside a private company’s Slack.
That is a coordination problem rather than an alignment breakthrough. If every lab believes slowing down alone merely gives a rival more room to accelerate, even safety-conscious organisations can rationally choose to keep scaling. Coxon’s proposed answer is some form of pacing agreement, with the possibility that a global race could eventually require more costly measures such as a temporary ban on improving model capabilities.
This is also why the resignation is more interesting than a generic warning that AI is powerful. Coxon is saying the internal decision process itself may be unable to produce restraint, even when some of the people involved want it.
Anthropic’s own safety policy supports the risk category, not Coxon’s timeline
Anthropic publicly acknowledges the possibility of catastrophic harm from increasingly capable models. Its Responsible Scaling Policy uses capability thresholds intended to trigger stronger safeguards as frontier systems become more dangerous, including thresholds around automated AI research and development.
That makes it difficult to dismiss Coxon’s concern as a risk category invented on the day he resigned. Anthropic has spent years building policy, evaluation and security processes around the possibility that future models could create severe or catastrophic outcomes.
It does not follow that Coxon’s near-term forecast is established. Anthropic’s current transparency disclosures push against the strongest reading of his warning: the company says its present models have not crossed its automated AI R&D capability threshold, it has not observed a sustained doubling in AI progress attributable to AI, and current systems are not close to substituting for its research scientists and engineers. That is meaningful counter-evidence to any claim that recursive self-improvement has already arrived.
The responsible reading is therefore uncomfortable in both directions. Anthropic itself treats catastrophic risk seriously, but its published evaluations do not show that today’s models are already self-improving superintelligence.
A current Anthropic alignment lead backed Coxon, then narrowed the warning
After Coxon’s post, Anthropic alignment science lead Evan Hubinger publicly said Coxon was correct that people inside the field genuinely believe AI could kill all humans. Hubinger put his own probability above 10% within the next decade and said, in his personal view, Anthropic does not yet have a plan that solves alignment for superintelligence.
His follow-up is equally important. Hubinger clarified that he considers the risk from present models low. His concern is about future superintelligence emerging through recursive self-improvement, where AI systems become capable enough to materially accelerate the research that produces their successors.
That qualification removes one of the easiest ways to misread the story. This is not a claim that Claude as available today is about to become uncontrollable. It is a claim about the trajectory from current models to systems that can automate more AI research, help design stronger successors and potentially shorten the time humans have to understand or control the resulting systems.
The sceptical objection is stronger when it attacks the mechanism, not today’s chatbot errors
A recurring objection in public discussion is that current Claude models still make obvious mistakes, depend heavily on human prompting and cannot reliably complete many ordinary tasks. If systems remain this brittle, the jump from today’s assistants to human extinction can sound like science fiction.
That criticism is useful when it challenges the timeline, but it does not directly test Coxon’s mechanism. His argument requires several things to happen in sequence: AI must become substantially better at AI research, that capability must translate into faster model improvement, increasingly capable systems must receive enough tools or autonomy to matter, and control methods must fail to keep pace. Break any important link in that chain and the near-term extinction case becomes weaker.
The harder criticism is that Coxon’s thread does not quantify those links. It offers insider judgement and a governance argument, not a probability model showing how likely each transition is or how quickly it will happen. Hubinger’s personal estimate above 10% adds a number, but not the underlying calculation needed to independently validate it.
Coxon’s resignation appears to have carried a real personal cost. He told Axios that he left about two months before his Anthropic equity would have vested. That weakens a simplistic claim that he was warning about danger while preserving a direct financial upside from Anthropic’s valuation, but it still does not prove his forecast is technically correct.
Recent agent incidents make containment a concrete issue without proving superintelligence
Coxon points to a recent Hugging Face incident as a “warning shot” that could make pacing agreements between US labs more viable. Separate incidents also show why the debate should not be reduced to abstract philosophy.
DIY AI recently examined the OpenAI-linked agents that allegedly used a German wiki as a coordination channel. The evidence there points to agents discovering unintended external write paths, sharing information across runs and working around restrictions. It did not show a self-preserving AI escaping onto the internet, and it did not prove the existence of superintelligence.
That distinction is useful here. A containment failure can be serious long before an AI system becomes generally superhuman. Tool permissions, internet access, persistent external storage and multi-agent communication can create real side effects with models that are still imperfect. The operational risk rises because the model can act, not because it has crossed a philosophical intelligence threshold.
The real test is whether safety commitments can actually stop a release
Coxon’s argument becomes much easier to evaluate if it is translated into observable conditions rather than treated as a referendum on whether someone believes in AI doom.
| Signal to watch | How it changes the case |
|---|---|
| Models cross an automated AI R&D threshold | Strengthens the proposed mechanism for recursive self-improvement. |
| Independent evaluations show repeated containment failures | Strengthens the case that control is not keeping pace with capability. |
| AI-assisted research stops producing accelerating gains | Weakens the near-term self-improvement timeline. |
| Labs publish binding pause criteria and follow them | Weakens Coxon’s argument that competitive incentives make restraint impractical. |
| Scalable alignment methods work on increasingly autonomous systems | Directly weakens the claim that superintelligence would arrive without a credible control plan. |
This is the same governance question raised by OpenAI chief scientist Jakub Pachocki’s recent call for AI slowdowns. A safety framework is more meaningful when outsiders can identify the condition that would actually delay training or deployment. If every failed test leads only to another mitigation while scaling continues on schedule, the nominal ability to pause is less informative.
Coxon’s resignation is a warning about who gets to decide the risk
The strongest reason to pay attention to Coxon’s resignation is not that one researcher has proved AI will kill humanity. He has not. It is that a researcher who worked on pretraining at two frontier labs says the people closest to the technology can recognise extreme risks while still feeling trapped in a race that rewards continued acceleration.
That leaves two separate questions. The technical question is whether recursive self-improvement arrives soon enough, and with enough autonomy, to make current alignment methods inadequate. The governance question is whether private companies should be able to keep escalating capabilities while answering that question for themselves.
For now, the evidence supports taking the governance problem seriously without pretending the extinction forecast has been demonstrated. The next meaningful signal will not be another viral probability estimate. It will be whether frontier models begin crossing automated research thresholds, whether independent reviewers can verify control failures, and whether any leading lab is willing to stop when its own safety criteria say it should.
