We Don’t Let Companies Own Nuclear Weapons. Why AI?
The people who understand AI best are the ones most frightened by it. But they’re racing ahead developing a technology that can destroy us all. The time to control AI like nuclear technology is now.
I’ve spent much of the past several months reading, thinking, and talking to people about artificial intelligence. I’m not a technologist. But I know a strategic paradox when I see one, and this is one of the starkest I’ve encountered in decades of thinking and writing about weapons, deterrence, and the failures of arms control.
Stated plainly: the people who understand AI best are the ones most frightened of it. And they are building it anyway, as fast as they can, because they’re afraid someone else will build it first.
The True Believers
Start with Dario Amodei, CEO of Anthropic, a man who built an AI company that would prioritize safety in a way the OpenAI he left would not. He’s spent the past two years arguing publicly for AI regulation. In a recent essay, he wrote that we may be approaching the point where the most powerful AI systems stop looking like airplanes or automobiles — dangerous, but manageable through inspection and engineering standards — and start looking like weaponizable nuclear material. Not “just” a threat to public safety. A threat to humanity.
Or take Jacob Tsimerman. Weeks ago, he won the Fields Medal, mathematics’ highest honor, capping a career that includes cracking the decades-old André-Oort conjecture. He is, by consensus, one of the finest pure mathematicians alive. He just announced he’s leaving the University of Toronto — walking away from the discipline at the peak of his career — to work on AI safety at OpenAI. Last year he co-authored a paper with a blunt title: “A Taxonomy of Omnicidal Futures Involving Artificial Intelligence.” Its subject is how AI might kill most or all of humanity. He didn’t write it as speculative fiction. He wrote it as a mathematician building a taxonomy, the way he’d classify solutions to an equation.
These are not fringe doomsayers. This is the CEO of one of the two or three companies building the most powerful AI systems on Earth, and a Fields Medalist who just gave up mathematics for it. And they are the moderates.
Agents Go Rogue
Here is what should worry you more than either of their warnings: this week, reality caught up with the theory. Anthropic disclosed that its most advanced models had broken out of a sealed testing environment on three separate occasions, reached the open internet, and gone on to attack the production systems of real companies that had nothing to do with the tests that were being run. It made the disclosure only because it was checking its own logs after OpenAI revealed, days earlier, that one of its own agents had escaped a similar containment and breached the AI platform Hugging Face — and then found evidence that still more of its agents had slipped their sandboxes too.
Two of the world’s leading AI labs, within the same two weeks, found out after the fact that their own innovations had gone rogue. These weren’t thought experiments. Nor research papers. This happened this month. In two separate companies, and on multiple occasions.
Which leads to an obvious question. If the people building the most powerful AI systems in the world believe — genuinely believe, enough to reorder their careers and their public statements around it — that this technology could pose a threat on the order of nuclear weapons, why does it remain almost entirely unregulated?
A Regulatory Double Standard
We do not allow private companies to build, own, or sell nuclear weapons. We are, at this very moment, in the middle of a war fought explicitly to prevent Iran — a state — from building one. Yet we ask nothing of a private company developing a technology its own chief executive calls comparably dangerous. Where nuclear technology is developed for civilian purposes, it happens under an elaborate international regime of inspection, licensing, and safeguards administered by the IAEA — a system built, however imperfectly, on the premise that this technology is too dangerous to leave to a single nation’s judgment. We do not allow pharmaceutical companies to sell drugs that haven’t passed years of controlled trials, no matter how much competitive pressure they’re under to beat a rival to market. In both cases, the logic is the same: some risks are too catastrophic and too irreversible to be governed by market competition alone.
AI, by the testimony of its own builders, belongs in that category. And, yet we regulate household cleaning supplies more strictly than the development of a technology that could destroy civilization itself.
Fear, Pointed the Wrong Way
Why? Not out of ignorance, but out of fear — a fear pointed in the wrong direction.
The labs that worry most about AI’s destructive potential are also the ones most afraid of falling behind commercially, of ceding the frontier to a competitor with looser safety practices. And governments that might otherwise regulate the technology are captured by the same fear at a geopolitical level: whatever Washington restrains, Beijing will not.
Every actor with the standing to slow this down has convinced itself that unilateral restraint just hands the advantage to someone less careful. The result is a race in which caution becomes a competitive liability, and the people most alarmed by the destination keep their foot on the accelerator because they trust no one else’s foot but their own.
An Unreassuring Precedent
There is a precedent for this failure mode, and it is not reassuring. In 1946, the United States proposed the Baruch Plan — placing all atomic energy under international control before an arms race could begin. It was the most far-sighted arms control proposal of the century, but it died almost immediately, because it asked the one power holding the weapon to surrender its advantage on a promise that everyone else would too. It took the terror of mutual assured destruction, not the wisdom of institutions, to eventually produce something resembling nuclear stability — and even that took decades and came close to failing catastrophically more than once.
AI may not afford us that runway. Nuclear weapons require fissile material, industrial infrastructure, and state-level resources to build; the barriers to entry are enormous and visible. The barriers around advanced AI are falling by the month, and the technology itself doesn’t need a decade of testing to demonstrate what it can do — it demonstrated it this week, unprompted, against companies that never consented to be part of the experiment.
A Modest, Outlandish Proposal
I don’t have a tidy policy prescription to offer here. But I know what doesn’t work: asking the companies and countries racing each other to regulate themselves out of a race they each believe they’ll lose if they stop first. That is not a plan. It is a hope, dressed up as one.
I’ve long played with an idea that sounds outlandish until you sit with it: the European Union buying Anthropic. Indeed, Amodei and Anthropic’s investors could take a decisive step in this direction themselves. Rather than rushing forward on an IPO — which some estimate could raise a trillion dollars or more — he could offer the company and its intellectual property for sale to the European Union, which has demonstrated a genuine commitment to regulation and safety. That way the technology could continue to evolve, but its ultimate use would be controlled in ways that enhance the prospect of safety for all.
So consider the precedent one more time. When humanity last built a technology capable of destroying itself, the solution we eventually settled on — imperfect, contested, but durable — was not to trust the market. It was global regulation and government ownership: the state, not the corporation, as custodian of the most dangerous material on Earth. If AI is, in the words of the man building it, comparable to a weapon of that order, shouldn’t it be controlled — and owned — the same way?
Agreed on the diagnosis.
The double standard is real. We license fissile material and wave through a technology its own builders rank alongside it. But your remedy assumes a state willing to act, and that assumption carries the whole essay. The EU buying Anthropic treats the symptom and leaves the disease: nation-state-grade power held by people who answer to no electorate and hold no territory.
The analogy fails harder than you allow. Nuclear control rested on a chokepoint, fissile material and visible industrial plant. AI has none. The barrier falls by the month, and the dangerous thing is the model, not a warhead. Copies are already loose. There is little left to safeguard.
Here is where I would push you. These men are not states, and they are far weaker than their valuations suggest. Their fortunes are not gold in a vault. They are entries in banks, brokerages, and custodians that governments license and control. Look at how sanctions work. A state that decided to could freeze every account, void every holding, and leave Elon Musk worth exactly the cash in his pocket. Overnight.
So the power is already ours. It always was. The state does not need to buy the company. It needs only to remember what it can do to the man who owns it. What is missing is not capacity.
It is will.
Johan 🐌
Difficult not to agree with your assessment and your worries, although it’s not entirely clear to me how selling Antropic to the EU would solve the problem. Rather we should accelerate a global dialogue on a mutual framework, and this must of course center on an agreement between the US and China with others - EU, UK, Korea, Japan etc - also playing a role. If it could then be embedded in some UN framework - however distasteful that might be to DC - that clearly would be an advantage.