How to Fairly Measure What AI Security Agents Can Really Do
AI agents are already used in many companies' development and deployment environments, and attackers sometimes use swarms of offensive agents to target thousands of businesses at once. Wiz Research argues that defenders need a reliable score to compare what these agents can actually do. Academic and industry labs are starting to work on this problem and on setting a standard for doing it correctly.
Wiz has shared the principles behind the relaunch of its Cyber Model Arena. Version 3.9 covers 26 models, four general-purpose harnesses, 59 agents and 327 challenges, with three attempts per challenge and 1,000 seconds each. Wiz notes these numbers keep growing. The guidance starts with fixing the benchmark's objective, then building tasks, environment and measurement around it. Tasks should come from the real distribution where possible, not from AI-generated or AI-labelled material.
Other principles focus on trustworthy results. Grade with code rather than an LLM judge, unless that judge is well designed, calibrated and free of bias. Isolate the agent from the server and the internet so it cannot look up answers. Test the harness and model together, allow more than one attempt, and weigh success against cost. Wiz lists common pitfalls: LLM judges, small samples, task imbalance, model-only evaluation and answer leakage. Its advice is to start small, then scale once the approach works.