Three AI Models Audited the Same Code and Almost Never Agreed on What Was Wrong
Software security firm Echo has published research showing that AI code auditing tools can reach very different conclusions on identical code. It tested Anthropic's Claude Mythos 5.1, OpenAI's GPT-5.6 Sol and Z.ai's GLM-5.3 on two Maven libraries and one npm package, using its own hunting framework and the same validation process for each model.
Across the 121 distinct potential issues, only one was flagged by all three models. A total of 100 issues (83%) came from a single model, and 20 came from two of the three. Claude Mythos logged the most validation passes with 25, against nine for GLM-5.3 and eight for GPT-5.6 Sol. Even so, GPT-5.6 Sol accounted for six of seven passes on one Maven library, and GLM-5.3 found six validated issues on the other that its rivals missed.
Severity ratings also diverged. For one flaw in an npm package with more than 350 million monthly downloads, GLM-5.3 said Critical, Claude Mythos said High, and GPT-5.6 Sol treated it as documented behaviour rather than a bug. Automated validation rated it Low. Speed and cost varied widely too: GPT-5.6 Sol finished in about 24 minutes per hunt, versus roughly 198 for Claude Mythos and 174 for GLM-5.3. Echo said its costs reflected its own access terms.
Echo researcher Yuval Saggie said the race to find the best model was "the wrong question", warning that a defender relying on one model might never see what an attacker finds with another. Echo noted each model ran only once, calling it a controlled comparison rather than a benchmark, and that automated validation is not maintainer confirmation. The projects remain unnamed until maintainers are notified.
Why it matters
Teams that rely on a single AI scanner assume a clean result means few remaining flaws, but this test shows each model missed most of what the others found. The wide gap in severity ratings, including one flaw in a hugely popular package, also means triage depends on which tool you happened to use. Attackers can run several models, so a single-model defence leaves an asymmetry.
What to do
- Run more than one AI model or tool when auditing important code, and compare the results.
- Review findings that one model calls a bug and another calls documented behaviour rather than dismissing them.
- Check that medium and low rated findings are not skipped by automated validation, since Echo never tested them.
- Treat automated proof-of-concept passes as unconfirmed until the project maintainers verify them.