How Auditors Could Check Whether AI Agents Are Really Being Watched

METR, a nonprofit that evaluates frontier AI, has set out how an assessor could investigate the control and monitoring of AI agents used inside AI developers. These "control assessments" ask whether an agent could cause serious harm without being stopped by the developer's security and oversight measures. Developers currently rely heavily on monitors to catch potentially harmful AI behaviour, so the proposal looks at how to gather evidence that those monitors are reliable. METR currently lists five core sub-questions and says it will update them as new capabilities and misaligned behaviours emerge.

The focus is on monitoring of AI agents by large language models (LLMs). METR notes that where an agent only needs a limited, known set of actions, standard cybersecurity measures such as limiting permissions through sandboxing may be enough. But many agents need broad permissions. An agent with internet access, for example, can take an enormous range of actions, and spotting harmful ones often requires understanding the wider context. LLM-based monitoring is therefore described as a critical part of automated oversight at scale.

Monitoring only helps if it is actually running on the relevant activity. The assessor would estimate what fraction of internal agentic activity is covered, weighting by risk: agents in tightly secured sandboxes need less oversight than those with internet access or access to internal credentials and infrastructure. This would require inference logs, monitor logs to cross-reference them, and employee interviews to find every source of AI activity.