Threat Intelligence

Choosing an AI Pentesting Vendor: Test It on Your Application, Not Theirs

The Hacker News · 18 Nov 2026
Key Takeaway Before buying AI pentesting, run a bake-off on your own application and insist on validated, reproducible evidence rather than a list of suspected issues.

Google's Threat Intelligence Group has reported, for the first time, a threat actor using a zero-day exploit it believes was developed with AI: a two-factor authentication bypass rooted in a logic flaw that conventional fuzzing and static analysis can easily miss. The article argues that agentic pentesting, where AI agents probe applications on their own, can help close that gap. It cites a survey in which 87% of US security leaders are planning, piloting or already using it.

The hard part is choosing the agent. The article describes three agents that all demo well: one stops at the login page, one cannot say what its coverage actually is, and one exploits real issues but has no hard limits for production. Each looks impressive on the vendor's own target, and the differences only show on yours. The suggested answer is a structured bake-off before signing.

Two questions stand out. The first is about context. Every tester must learn your site before testing, and in CyCognito's model recon and analysis take 20 to 30 of a manual pentest's 63 to 95 hours. Some agents rebuild that map from a blank URL each run, while others keep a model of your site current. A starting map should be a head start, not a boundary. The second is about proof. A suspicious pattern is not an exploitable vulnerability, so ask whether the agent exploits what it suspects, reproduces findings and supplies evidence your team can verify.

The article offers an example. In an authorised assessment of a major insurer's production website, an AI agent found that the one-time passcode request could send login codes to any email address. It proved this by logging in with a code sent to its own inbox, then guardrails stopped the test. The insurer said it had pentested the site about six months earlier. The full checklist has 15 questions, and this excerpt covers only part of it.

Why it matters

Organisations relying on periodic manual pentests or scanners may have logic flaws, such as authentication and passcode weaknesses, that those methods miss, as the insurer example shows after a test only six months earlier. For anyone buying AI testing, a good demo breaks the assumption that capability transfers to your app: login access, validation of findings and production guardrails must be proven on your environment.

What to do

  • Run any shortlisted AI pentesting vendor against your own application, not their demo target.
  • Ask whether the agent can reach your logged-in workflows and test across user roles.
  • Require reproducible evidence for each finding, not just a list of suspected issues.
  • Confirm what hard limits the agent follows when testing production.
AI pentesting web application security vendor selection 2FA bypass penetration testing

Summarised by CISO AI from The Hacker News, written with Claude Sonnet 5.5. We link back to every original so you can read it yourself.