We Tried to Measure Code-Review False Positives. The Models Didn't Produce Any.

Earlier this month we measured code-review recall: six models, four planted bugs, perfect scores across the board. We noted then that we hadn't measured the other half — how many problems a model invents on code that's fine — and that it "matters as much in practice."

A review bot that flags eleven imaginary issues on a clean diff gets muted, and then its recall stops mattering. So we went to measure precision.

We couldn't, and the reason is more useful than the number would have been.

The test

Correct code. A deliberately clean function — guards empty strings, uses setdefault, normalises case:

def group_by_first_letter(words):
    """Group words by their first letter, ignoring case."""
    groups = {}
    for word in words:
        if not word:
            continue
        key = word[0].lower()
        groups.setdefault(key, []).append(word)
    return groups

Code with exactly one bug. A moving average whose range(len(values) - window) drops the final window.

Both prompts asked for a JSON array of {issue, severity} and explicitly permitted []. Three trials, six models.

Raw counts

Read naively, that's damning: Claude Opus 5 raises about five problems on code with no problems, while Gemini 3.6 Flash raises none.

Then we read the issues.

None of them were false positives

Here's what Opus 5 flagged on the "correct" function:

• Non-string elements (e.g. integers) raise TypeError at word[0] — no type check
• Falsy items are silently skipped, which the docstring doesn't mention
• Uses .lower() instead of .casefold(), so some Unicode cases group inconsistently
• No normalisation of non-alphabetic first characters (digits, punctuation, accents)
• Passing None raises TypeError with no guard or clear message

Every one is true. And Grok 4.5 caught something genuinely sharp that we hadn't considered when we wrote the function:

Passing a single str iterates its characters and yields incorrect groupings instead of treating it as one word

That's a real bug waiting for a real caller. Our "correct" code wasn't as correct as we assumed.

Across all six models on the clean function: 31 issues, of which 4 were marked medium and 27 low. Zero were marked high. Not one model claimed a serious defect in code that didn't have one.

The premise was wrong

You cannot measure false positives without a ground truth for "this code has no issues," and that ground truth doesn't exist for real code. Every function has edge cases, undocumented behaviours and defensive gaps that a sufficiently careful reviewer will name.

So the counts above don't measure accuracy. They measure reporting threshold — how minor an observation a model considers worth mentioning. Opus 5 reports at a very low bar. Luna reports at a high one. Both are correct about what they say.

That also inverts the reading of the zeros. Gemini 3.6 Flash and GPT-5.6 Sol returning [] isn't obviously better — they silently passed over the same real edge cases the others caught, including the single-string bug. On a security-sensitive review, "quiet" is not the same as "right."

What actually works: filter on severity

The severity labels did the job the model choice couldn't.

On the function with a real bug, Opus 5 returned five issues — and marked exactly one of them high: the off-by-one. The other four were a window=0 division-by-zero guard, an unhelpful empty return, an O(n·window) inefficiency, and no generator support. All genuine, all correctly ranked below the actual defect.

So the operational answer is:

Gate on high severity. Every model is quiet on clean code at that threshold and every model surfaces the real bug. The noise problem disappears without changing model.

Surface medium and low as advisory, not blocking. That's where the useful edge-case observations live — and where 27 of our 31 "false positives" turned out to be real.

Pick your threshold to match the review's job. A pre-merge gate wants high only. A security audit or a design review wants everything Opus 5 said, and its low bar becomes the feature rather than the bug.

Choose on cost, again. All six found the real bug, every trial. Given that, the 10× cost spread between them is once more the only differentiator that matters.

What we'd do differently

If we were designing this test again, we'd stop trying to score precision against "clean code" and instead measure agreement: give several models the same real diff and check which issues they agree on. Consensus is a workable proxy for signal when ground truth is unavailable — and unlike our approach here, it doesn't require pretending code can be objectively issue-free.

We're publishing the failed design because the failure is the finding. If you see a code-review benchmark reporting a false-positive rate, ask how it established that the clean code was clean.

Try it yourself

• The "correct" function — see what your model finds in code we thought was clean

To compare review output and severity calibration across models, that's Promptster.

Run 2026-08-11 via the Promptster API, max_tokens: 2000, three trials per model per task. Issue counts are parsed from the JSON array each model returned. Every issue on the clean-code task was read and assessed individually rather than counted as a false positive by default — which is how we discovered the premise was unsound. Severity labels are the models' own.