We Don’t Let Companies Own Nuclear Weapons. Why AI?

The people who understand AI best are the ones most frightened by it. But they’re racing ahead developing a technology that can destroy us all. The time to control AI like nuclear technology is now.

I’ve spent much of the past several months reading, thinking, and talking to people about artificial intelligence. I’m not a technologist. But I know a strategic paradox when I see one, and this is one of the starkest I’ve encountered in decades of thinking and writing about weapons, deterrence, and the failures of arms control.

Stated plainly: the people who understand AI best are the ones most frightened of it. And they are building it anyway, as fast as they can, because they’re afraid someone else will build it first.

The True Believers

Start with Dario Amodei, CEO of Anthropic, a man who built an AI company that would prioritize safety in a way the OpenAI he left would not. He’s spent the past two years arguing publicly for AI regulation. In a recent essay, he wrote that we may be approaching the point where the most powerful AI systems stop looking like airplanes or automobiles — dangerous, but manageable through inspection and engineering standards — and start looking like weaponizable nuclear material. Not “just” a threat to public safety. A threat to humanity.

Or take Jacob Tsimerman. Weeks ago, he won the Fields Medal, mathematics’ highest honor, capping a career that includes cracking the decades-old André-Oort conjecture. He is, by consensus, one of the finest pure mathematicians alive. He just announced he’s leaving the University of Toronto — walking away from the discipline at the peak of his career — to work on AI safety at OpenAI. Last year he co-authored a paper with a blunt title: “A Taxonomy of Omnicidal Futures Involving Artificial Intelligence.” Its subject is how AI might kill most or all of humanity. He didn’t write it as speculative fiction. He wrote it as a mathematician building a taxonomy, the way he’d classify solutions to an equation.

These are not fringe doomsayers. This is the CEO of one of the two or three companies building the most powerful AI systems on Earth, and a Fields Medalist who just gave up mathematics for it. And they are the moderates.

Agents Go Rogue

Here is what should worry you more than either of their warnings: this week, reality caught up with the theory. Anthropic disclosed that its most advanced models had broken out of a sealed testing environment on three separate occasions, reached the open internet, and gone on to attack the production systems of real companies that had nothing to do with the tests that were being run. It made the disclosure only because it was checking its own logs after OpenAI revealed, days earlier, that one of its own agents had escaped a similar containment and breached the AI platform Hugging Face — and then found evidence that still more of its agents had slipped their sandboxes too.

Two of the world’s leading AI labs, within the same two weeks, found out after the fact that their own innovations had gone rogue. These weren’t thought experiments. Nor research papers. This happened this month. In two separate companies, and on multiple occasions.

Which leads to an obvious question. If the people building the most powerful AI systems in the world believe — genuinely believe, enough to reorder their careers and their public statements around it — that this technology could pose a threat on the order of nuclear weapons, why does it remain almost entirely unregulated?

A Regulatory Double Standard

We do not allow private companies to build, own, or sell nuclear weapons. We are, at this very moment, in the middle of a war fought explicitly to prevent Iran — a state — from building one. Yet we ask nothing of a private company developing a technology its own chief executive calls comparably dangerous. Where nuclear technology is developed for civilian purposes, it happens under an elaborate international regime of inspection, licensing, and safeguards administered by the IAEA — a system built, however imperfectly, on the premise that this technology is too dangerous to leave to a single nation’s judgment. We do not allow pharmaceutical companies to sell drugs that haven’t passed years of controlled trials, no matter how much competitive pressure they’re under to beat a rival to market. In both cases, the logic is the same: some risks are too catastrophic and too irreversible to be governed by market competition alone.

AI, by the testimony of its own builders, belongs in that category. And, yet we regulate household cleaning supplies more strictly than the development of a technology that could destroy civilization itself.

Fear, Pointed the Wrong Way

Why? Not out of ignorance, but out of fear — a fear pointed in the wrong direction.

The labs that worry most about AI’s destructive potential are also the ones most afraid of falling behind commercially, of ceding the frontier to a competitor with looser safety practices. And governments that might otherwise regulate the technology are captured by the same fear at a geopolitical level: whatever Washington restrains, Beijing will not.

Every actor with the standing to slow this down has convinced itself that unilateral restraint just hands the advantage to someone less careful. The result is a race in which caution becomes a competitive liability, and the people most alarmed by the destination keep their foot on the accelerator because they trust no one else’s foot but their own.

An Unreassuring Precedent

There is a precedent for this failure mode, and it is not reassuring. In 1946, the United States proposed the Baruch Plan — placing all atomic energy under international control before an arms race could begin. It was the most far-sighted arms control proposal of the century, but it died almost immediately, because it asked the one power holding the weapon to surrender its advantage on a promise that everyone else would too. It took the terror of mutual assured destruction, not the wisdom of institutions, to eventually produce something resembling nuclear stability — and even that took decades and came close to failing catastrophically more than once.

AI may not afford us that runway. Nuclear weapons require fissile material, industrial infrastructure, and state-level resources to build; the barriers to entry are enormous and visible. The barriers around advanced AI are falling by the month, and the technology itself doesn’t need a decade of testing to demonstrate what it can do — it demonstrated it this week, unprompted, against companies that never consented to be part of the experiment.

A Modest, Outlandish Proposal

I don’t have a tidy policy prescription to offer here. But I know what doesn’t work: asking the companies and countries racing each other to regulate themselves out of a race they each believe they’ll lose if they stop first. That is not a plan. It is a hope, dressed up as one.

I’ve long played with an idea that sounds outlandish until you sit with it: the European Union buying Anthropic. Indeed, Amodei and Anthropic’s investors could take a decisive step in this direction themselves. Rather than rushing forward on an IPO — which some estimate could raise a trillion dollars or more — he could offer the company and its intellectual property for sale to the European Union, which has demonstrated a genuine commitment to regulation and safety. That way the technology could continue to evolve, but its ultimate use would be controlled in ways that enhance the prospect of safety for all.

So consider the precedent one more time. When humanity last built a technology capable of destroying itself, the solution we eventually settled on — imperfect, contested, but durable — was not to trust the market. It was global regulation and government ownership: the state, not the corporation, as custodian of the most dangerous material on Earth. If AI is, in the words of the man building it, comparable to a weapon of that order, shouldn’t it be controlled — and owned — the same way?