Why AI audits hallucinate, and what actually fixes it

If you have run a smart contract through an LLM and received forty confident findings, you already know the problem. The report is not obviously wrong — every item cites real line numbers, uses correct terminology, and describes a plausible vulnerability. Checking them takes longer than the audit saved.

Why it happens

Ask a model to find vulnerabilities and you have framed a task whose implied success condition is finding some. A model that returns "this contract looks fine" has, from the perspective of the instruction it was given, done nothing. So it produces findings — and because it has read an enormous number of audit reports, it produces findings that look exactly like real ones.

The failure is not ignorance of security. The model genuinely knows what reentrancy is. The failure is that pattern-matching a shape ("an external call near a state write") is much easier than verifying the specific claim ("this call, on this line, can re-enter this function given these modifiers"). The first produces a fluent paragraph. Only the second is an audit.

What does not fix it

• —Prompting harder. "Only report real issues" and "be conservative" shift the volume a little. They do not change the incentive, because the model has no way to check its own claim while it is making it.
• —A bigger model. Better models write more convincing false positives. Fluency rises faster than verifiability.
• —Static analysis alone. Slither and Mythril are precise about the bug classes they encode and silent about everything else — the business-logic flaws that cause most real losses are outside what they model at all.

What does: making the model argue against itself

The asymmetry that causes the problem also solves it. Generating a plausible finding is easy; defending a specific finding against the actual code is hard. So Weakspot runs a second pass whose only job is to attack the first one: each candidate finding is re-read against the exact code it cites, in a fresh context, and asked whether the claim survives. Findings that cannot be substantiated are dropped before the report exists.

The effect on one internal test run, on a contract set with known planted bugs:

first pass (generate)   48 findings
second pass (verify)     7 findings
actually real            1 finding

Two honest observations about those numbers. The verify pass removed roughly 85% of the output — that is the difference between a report you skim and discard and one you read. And it did not reach perfect precision: 7 surviving findings for 1 real bug is still work for a human. The claim is that the second number is a usable artifact and the first is not, which is a much smaller claim than "AI can audit your contract".

How to evaluate any AI auditor, including this one

• —Run it on code you have already audited. You know the answer key. Count what it invents, not what it catches.
• —Run it on deliberately clean code. A tool that finds four criticals in a correct contract is telling you what its findings are worth. Weakspot keeps clean fixtures in its own test set for exactly this reason.
• —Read the highest-severity finding closely. Precision at the top of the list is what determines whether the tool saves you time.