New Benchmark Puts AI Security Agents to the Test

As businesses increasingly explore AI-powered security tools, a new challenge has emerged: how do you actually know if an autonomous security agent is any good? According to The Hacker News, these AI agents are becoming skilled at finding software bugs, but until now there has been no reliable way to verify their claims. An agent's report might describe a genuine exploit, a near miss, or something entirely invented, and the write-up often looks identical in each case.

Manually checking every claim against a real target takes a security expert a full day per test run, a process that becomes unworkable once teams need to compare multiple AI models and configurations across many repeated tests. Reports also fail to show what an agent never attempted to check, and they don't flag risky behaviour, such as an agent deleting data or revoking access keys while chasing a result.

To solve this, CTF.ae built XRanges for AI, a platform that deploys realistic, fully instrumented target systems so it can record exactly what an agent does and score each run using four independent measures. The tool is designed to give AI development teams and security buyers an honest, consistent way to compare how different agents actually perform, rather than relying on self-reported summaries.