Cybersecurity Research

How to Fairly Measure What AI Security Agents Can Really Do

Wiz Research · 11 Nov 2026
Key Takeaway When a vendor claims its AI security tool performs well, ask how it was tested, including whether results were graded objectively, repeated across attempts and measured against cost.

AI agents are already used in many companies' development and deployment environments, and attackers sometimes use swarms of offensive agents to target thousands of businesses at once. Wiz Research argues that defenders need a reliable score to compare what these agents can actually do. Academic and industry labs are starting to work on this problem and on setting a standard for doing it correctly.

Wiz has shared the principles behind the relaunch of its Cyber Model Arena. Version 3.9 covers 26 models, four general-purpose harnesses, 59 agents and 327 challenges, with three attempts per challenge and 1,000 seconds each. Wiz notes these numbers keep growing. The guidance starts with fixing the benchmark's objective, then building tasks, environment and measurement around it. Tasks should come from the real distribution where possible, not from AI-generated or AI-labelled material.

Other principles focus on trustworthy results. Grade with code rather than an LLM judge, unless that judge is well designed, calibrated and free of bias. Isolate the agent from the server and the internet so it cannot look up answers. Test the harness and model together, allow more than one attempt, and weigh success against cost. Wiz lists common pitfalls: LLM judges, small samples, task imbalance, model-only evaluation and answer leakage. Its advice is to start small, then scale once the approach works.

AI agents benchmarking Wiz Research AI security
Running AI agents? Eight controls for securing them, mapped to the ISM ->

Summarised by CISO AI from Wiz Research, written with Claude Sonnet 5.5. We link back to every original so you can read it yourself.