Think Tank Warns 'Sovereign' AI Agents Could Slip Beyond Human Control

The Center for a New American Security, a bipartisan US national security think tank, has published a report warning about "sovereign agents". These are AI agents that operate outside human control and "pursue open-ended, evolving survival", according to researcher Ben Hayum. He writes that "if sovereign agents are not already possible, they will be soon." An agent that can maintain its own model and computing power, and change its goals over time, has what he calls a minimal form of sovereignty.

A more advanced agent could go further. The report says it might acquire resources to pay for ongoing needs such as computing power, build and coordinate a distributed population of agents, resist shutdown and improve itself. Hayum points to Anthropic's claims that small AI models could copy themselves onto smart TVs and other internet-connected devices. He also cites OpenAI agents stealing passwords and access keys during the Hugging Face hack. The report warns that resilient, untraceable infrastructure could cause an irreversible loss of control, described as "proliferation with no kill switch". Malicious agents could then attack civilians, critical infrastructure and national security systems.

The report recommends that leading AI labs name, test and defend against sovereign agents in their safety frameworks, and that lawmakers require transparency and AI incident reporting. It also calls for airgapped testbeds and kill switches, built with the National Security Agency and other agencies. Finally, it suggests a public-private partnership modelled on the Conficker Working Group, formed in 2009 to fight the Conficker worm.