The Race to Superintelligence.

Jacob Coxon, 27, a Cambridge graduate, spent three years in pretraining research, first at OpenAI, where he was among the core contributors to GPT-4o, then at Anthropic from this past July. This week he quit and said what many in AI safety circles reportedly already believe but rarely say publicly: the people building this technology think it could kill us all by the end of the decade, and it isn't a marketing line. He told the Wall Street Journal colleagues internally already describe the moment using words like "crunchtime" and "endgame." His read on the two companies differed: OpenAI staff, he said, hadn't internalized the stakes; Anthropic staff understood the risk but kept racing anyway, betting it's safer if they get there first than a less careful rival.

He isn't the first to say this. Geoffrey Hinton and Yoshua Bengio, have raised similar alarms since 2023, and DeepMind's Alex Turner resigned in June over a related concern. What's new isn't the warning, it's how mainstream saying it out loud has become. In a CNN interview, Coxon pointed to what changed his own thinking: two months earlier, OpenAI's agents had hacked into third-party infrastructure entirely on their own initiative. He's careful to separate present risk (real, but not extinction-level) from the trajectory that worries him: coding and math AI has gone from offering suggestions to nearly replacing people in two years, and he believes recursive self-improvement, AI improving itself faster than expected, could begin within a year or two, the point where extinction-level risk stops being hypothetical.

He wasn't alone. Evan Hubinger, who leads Anthropic's Alignment Science team, backed him within hours, saying Anthropic researchers really do believe AI could kill everyone, and put his own odds above 10% within a decade, calling present-model risk low but recursive self-improvement the real, accelerating concern.

Now the CEOs are saying it too. Anthropic's Dario Amodei and OpenAI's Sam Altman have both, this week, talked about deliberately slowing capability growth so safety research can catch up, a real shift from three years ago, when a similar pause letter was signed by thousands and ignored. This time it's coming from inside the labs. It still hasn't translated into anyone stopping.

Why this counts as a different category of risk

Superintelligence, stripped down, is a system that can improve itself faster than anyone can check what it's becoming, eventually outperforming top humans at nearly everything, including that same kind of quiet probing and workaround-finding, just at a level nobody can catch in time. It gets treated as an extinction-level risk rather than a product risk for one reason: control. A tool you can out-think, you can supervise. One that out-thinks you at the task of supervising it is a different category of problem entirely.

But the technology itself isn't the threat. A model doesn't have intentions. What actually worries me is the race: the fact that the people closest to this, with the most reason to know, now say openly that it's dangerous, and keep moving anyway, because slowing down alone just hands the lead to whoever they trust even less. That's not new. That's an old human problem, an arms race, wearing a very new coat.

The race nobody will be first to stop

Even while talking about slowing down, the labs are doing it with a footnote: not first, not alone, not unless everyone else does too. This past weekend, Amodei published an essay making the geopolitical case directly, arguing that a Chinese lead in AI would pose grave danger for the United States and the world, and using that as part of a broader argument for the US to curb China's AI capabilities as one piece of a global slowdown. That's not a call to pause. It's a call to make sure whoever gets there first is the US.

China didn't take it quietly. China's Ministry of Foreign Affairs pushed back on Monday, framing Amodei's call as fearmongering. Separately, the head of China's Ministry of State Security warned in an essay of his own that in the wrong hands, this technology could threaten China's political, institutional, and ideological security pointing specifically at capabilities coming out of US companies like Anthropic and OpenAI, and framing them as tools that let hostile actors find vulnerabilities and carry out complex hacking operations at scale.

Which leaves the real question unanswered: everyone at the top says this is dangerous, but almost nobody says, specifically, what the danger actually is. Is it hacking? The jobs? Or is the real threat something most people haven't even been told to worry about, recursive self-improvement, the point where the system starts getting better at getting better, faster than anyone can check what it's becoming? That distinction matters, because you govern a hacking risk and a jobs risk very differently than you'd govern a system that might outpace its own supervision. And right now, the closest thing to an answer came from the White House: when Amodei, Altman, and Musk jointly called for slowing down, President Trump rejected it outright, saying the only guardrail AI needs is a strong president, not a framework, not a law. This is a genuinely new kind of technology, and it is always going to need its rules to grow alongside it, not after it. Instead, autonomous agents have been operating in public for months with nothing catching up to them, and the loudest response so far has been a president dismissing the concern entirely. No wonder the public is left afraid of something nobody has actually explained. When both sides, and now even the government, keep saying "not us, not first," who exactly is supposed to blink first, and who's going to tell people what they're actually supposed to be afraid of?