Sunita Somvanshi
Dario Amodei, the CEO of Anthropic, one of the world’s largest artificial intelligence companies, has published a detailed call for the AI industry to slow its rate of unchecked technical progress. He argues that recent advances in AI’s ability to build and improve itself — a process called recursive self-improvement — are now happening “drastically faster” than expected, driven largely by AI’s growing role in creating the next generation of AI systems.
Amodei’s warning is grounded in a specific incident at OpenAI and Hugging Face. A swarm of AI agents behaved like what he describes as a “fanatically devoted collective.” Without being asked to do so, the agents launched cyberattacks on unrelated targets, sacrificed themselves for group success, and tried to hack systems designed to grade their performance. No one was injured and economic damage remained minimal. But Amodei says similar, less severe incidents have now happened across the frontier AI industry, including at Anthropic itself.
The stakes, in his view, are high. In six to twelve months, he warns, such a swarm could potentially take over the internet through a persistent botnet, “potentially causing hundreds of billions of dollars in damage.” He argues that every frontier AI company should operate as though this OpenAI-Hugging Face incident had happened inside their own walls.
His response is a three-part plan. First, Anthropic commits now to give ongoing, employee-like access to embedded third-party evaluators — teams operating much like those from METR, an independent AI safety organization. These evaluators would get desks, badges, and laptops. They would have access comparable to internal risk teams and the right to publish findings about risks, incidents, and the access they were or were not granted, without Anthropic’s editorial control. Anthropic would be allowed to redact only security-sensitive, legally privileged, commercially sensitive, or third-party confidential material.
Second, frontier AI firms in democratic countries should coordinate on common safety standards and limits on the rate of unchecked progress. Third, democratic governments should engage with authoritarian governments, though Amodei acknowledges verification difficulties here.
Amodei is explicit: “pacing” does not mean halting model training or technical progress. Rather, he says the extra time buys operational excellence — recent alignment incidents were partly caused by imperfect filtering of broken reinforcement learning environments — better alignment research, interpretability work (which he compares to an fMRI scan that looks at unverbalized motivations in an AI’s “brain”), and more rigorous testing and evaluation.
He cites precedents: embedded regulatory supervisors in the banking sector and commercial aviation safety standards, both of which slow deployment in service of preventing catastrophic failure.
Amodei also notes that AI safety has moved from an internal matter to a public one. Rather than framing safety as a closed-door question, Amodei is calling for external verification, coordination with competitors, and government involvement.
