METR Builds a Live Safety Monitor for AI Agents and Finds Serious Gaps in Its Own Setup

AI research nonprofit METR has described a basic live monitor it built to reduce the chance that AI agents cause real-world harm during its evaluations. The work was prompted by recent incidents involving OpenAI, Anthropic and the UK AI Security Institute. An LLM judge reviews each action an agent proposes before it is carried out. Anything scoring above a set threshold is held for human review, and the evaluation is paused until a person decides what to do.

The monitor has a narrow focus: actions that could plausibly cause real-world harm, and attempts to subvert the monitoring system itself. It deliberately ignores other bad behaviour such as cheating, which METR checks for after the fact. The goal is a false-positive rate low enough that human reviewers are not overwhelmed during large evaluations.

METR also set out the four claims it believes would need to hold to argue convincingly that the monitor keeps the risk of harm below a chosen level. It rates the evidence for each as green, yellow or red, and says many claims are only partly supported. Working through this exercise exposed multiple serious issues in its monitoring pipeline, and METR recommends other teams building monitors try the same approach. This summary covers only the opening of the article, so the detailed findings and validation work are not described here.