Buying an AI Pentesting Tool? Test Whether It Proves Findings or Just Predicts Them

The term "AI pentesting" now covers very different products. Some platforms plan and run attack chains on their own and return exploitation evidence for every finding. Others are vulnerability scanners with a language model summarising the output. According to SOCRadar, on a datasheet or in a scripted demo against a vendor-prepared target, the two can look almost identical.

SOCRadar's 30 questions cover six areas: exploitation depth, autonomy and human oversight, evidence quality, production safety, data handling, and coverage. Most include a description of what a credible answer sounds like, so they can be used in vendor calls, RFPs and pilot scoping. The guiding test is simple: a credible tool proves things, while a weak one predicts things. A scanner says "this might be vulnerable", whereas a pentest says "we got in, here's how".

The opening questions advise buyers to ask to see a real exploited finding with its evidence attached, and to check that reports clearly label findings by risk level. Buyers should also ask for an end-to-end attack chain, and for examples of access control or business logic flaws the tool found on its own, since these rarely have CVE numbers. On autonomy, "fully autonomous" is not the goal. You want autonomy within limits your own team controls, with different settings for staging and production. Finally, ask what "human validated" really means. The only good answer to unproven findings is: no evidence, no finding.