My Local-LLM Code Review Benchmark Was 73% Ghosts

Tue, 15 Sep 2026 00:00:00 GMT

I keep one DGX Spark on a desk at home for my own projects, and I wanted something simple: a code review gate that runs on it, costs $0 per invocation, and catches real bugs before a paid frontier reviewer ever sees the PR. Local models are free once the hardware exists, so the only question left was whether they are actually any good at reviewing code, and the only answer I trust is one measured on my own code.

Over a few weeks, my Claude Code sessions and I benchmarked two local models against a golden set of seven real pull requests from one repo, the private monorepo behind this lab. (A six-repo set built from bugs that actually shipped comes later this week.) We ran well over a thousand review completions, found a result exciting enough to almost publish, and then audited our own measurement and watched most of it collapse. Total API spend for the entire program: under $10, nearly all of it on a frontier model used as a judge rather than a reviewer.

This is the story of the collapse, because the collapse is the useful part. Most of it had one cause: 73 of the 99 bugs in our ground truth were ghosts, already fixed inside their own pull requests before any reviewer of the final diff could have seen them. The single transferable lesson: your benchmark's ground truth deserves more scrutiny than your model, and it will not get it unless you schedule it.

The stack

Everything below runs on one NVIDIA DGX Spark (GB10, the ARM/Grace desktop box, 128 GB unified memory), with both models resident simultaneously under vLLM behind OpenAI-compatible endpoints:

Qwen3.8-27B, NVFP4-quantized, served at 262K context: the careful model, our "eyes." Slow (~11 tok/s decode), strongest local review signal.
Nemotron-3.5-Lightning-30B-A3B, also NVFP4: the fast model, our "hands." ~7× the decode speed, near-perfect structured output and tool-calling.
semgrep as the deterministic floor, with repo-specific rules for our known historical bug classes.
Claude Sonnet 4.6 over the API as the judge, used only to adjudicate eval results, at cents per decision. Every number below was judged by 4.6; the pinned-basis column was re-judged with Sonnet 5 before publication (appendix, "Judge vintage check").
CodeRabbit as the paid ceiling: it reviews every PR after the local gate has had its shot.
The gate itself is a small Python CLI (von review) that assembles the diff, fans out finder seeds, and runs a verifier pass. Thin plumbing.

No cluster, no cloud GPUs, no fine-tuning. The point of the program was to find out what this exact commodity setup can and cannot be trusted to do.

The instrument

The gate under test is a three-tier pre-push pipeline: a deterministic semgrep floor, a recall-first "bug hunter" finder sampled across seeds, and a skeptical verifier that reads the actual file and tries to refute each finding.

The instrument is a golden set: 8 real merged PRs, ground truth derived from what a frontier reviewer plus human curation actually flagged. Every configuration change to the gate has to beat its predecessor on this set before it ships. We call the rule eval-before-rewiring, and it is the only reason the rest of this story is measurement instead of vibes. It is also, as you'll see, not enough on its own.

Act 1: The instrument was lying (the first time)

The first overnight sweep compared prompt and sampling variants across both models, including a Qwen arm with thinking mode enabled, which we expected to win. It scored best (11 true positives vs 4 for every other arm), and we almost wrote it up as "reasoning helps."

Then we looked at the completions. Zero of 96 contained any thinking markup. The arm that "won with reasoning" had never reasoned at all.

The mechanism is worth knowing if you run local models: our harness requests strict JSON via response_format: json_schema. Under vLLM's guided decoding, with no reasoning parser configured server-side, the grammar clamps generation from the first token. The model is structurally incapable of writing into its think block: the request says "think carefully," and the decoder says "your next token is {." The chat template had injected its highest-effort reasoning instruction and pre-opened a think tag that could never be used.

So the sweep hadn't measured thinking. It had measured the instruction to think, with no thinking behind it. Pure framing. And on the strict scorer, that framing alone took recall from 0.058 to 0.164, at no latency cost.

Act 2: Real thinking lost at every effort level

The obvious objection: fine, framing beats suppressed thinking, but surely actual thinking beats both? We unclamped it and ran the full sweep: three effort levels, 96 runs each, 21.5 hours of wall clock, telemetry verifying the model genuinely reasoned this time.

arm (strict scorer)	true positives	recall	wall clock
framing only (no reasoning tokens)	11	0.164	~1.4h
true thinking, low effort	3	0.043	4.7h
true thinking, medium	1	0.014	6.6h
true thinking, xhigh	1	0.014	10.1h

The failure mode was visible in the transcripts: the model would spend thousands of reasoning tokens correctly enumerating candidate issues, then dismiss them. "Not severe enough to block." Verdict: pass, zero findings. For a recall-first task, deliberation bought conservatism, not coverage.

We were now sitting on a tidy, counterintuitive headline: framing beats thinking. Hold that thought, because the next two acts are about what happened when we tried to make that number trustworthy.

Act 3: The judge that agreed with us

Strict matchers (exact-ish file/line/category matching) are reproducible and free, but they undercount: a finding that describes the right bug with drifted line numbers scores zero. So we added the standard fix, a frontier model as matching judge: given ground truth and a reviewer's findings for the same PR, decide which findings describe the same underlying problem. Eighty-eight judge calls, thirty-five cents.

The calibrated numbers were glorious. The framing arm's true recall came back as 0.667 against 0.400 for the best alternative. It was the only arm credited with a critical catch. We drafted the article. An earlier version of this very piece stated those numbers as findings.

Then we did the thing that saved us, mostly by luck: we audited a sample of the judge's match decisions by hand before publishing.

Three out of fourteen sampled matches were real. The judge was theme-matching: a dead-code nit "matched" a sync-state bug because both lived in the same file; a finding that talked itself out of its own claim ("not a bug, drop this") still got credit for the bug it had retracted. We tightened the rubric. The tightened judge theme-matched too, just more politely. What finally held was a two-stage protocol: a mapper proposes matches, and a second adversarial pass must confirm each one by articulating the shared failure mode, with every decision persisted for audit. Under the confirmed judge, run twice for repeatability, the results were identical both times.

Act 4: The golden set was haunted

The confirmed judge collapsed the numbers, but auditing why it collapsed them exposed the deeper defect, and this one was in the ground truth itself.

Our golden set was built from a frontier reviewer's comments on real PRs. Those comments were made against intermediate commits. Authors then fixed most of those bugs inside the same PR, before merge. Our benchmark replayed each PR as one diff, base to final head, and asked reviewers to find the bugs. 73 of the 99 located ground-truth items, including both criticals, were fixed within their own PRs, absent from the code any reviewer ever saw. We had been grading models on their ability to find ghosts, and the lenient judge had been happy to certify sightings.

Corrected, on the findable basis (bugs actually present in the reviewed code): the celebrated framing arm caught 1 of 7. The best competing arm caught 0. The production three-seed deep configuration, the one we actually ship, caught 4 of 7 (0.571 findable recall). Those are the honest numbers, and they are the only numbers from this program I'd repeat at a dinner table.

The fix shipped as golden set v2: every ground-truth item is now pinned to the commit state its evidence was made against, findability is verified deterministically (the item's location must exist in the pinned diff), and a second ground-truth tier was mined from post-merge reality: bugs that actually shipped and later needed fixing, including one that silently corrupted 191 production database rows. Bugs reality found, as a separate scoring basis no judge can inflate.

Act 5: The headline fails replication

One question remained for the original finding: framing beat thinking on one model family, measured by a lenient judge. Does the effect survive the honest instrument, and does it transfer?

We ran the replication: five arms, 480 completions: framing on Nemotron, no framing on Nemotron, thinking-JSON on Nemotron, and two framing dose variants on Qwen (posture tag alone vs full framing words at low effort). Under the confirmed judge, across all five arms: one confirmed catch, total. On Nemotron, the unframed arm out-ranked both framed arms even on the strict scorer. The dose question dissolved: nothing survived confirmation to dose.

So the honest epitaph for our own headline: framing-beats-thinking was a within-family ranking artifact, measured against haunted ground truth by a lenient judge. What actually separates configurations is none of the prompt cosmetics: it's single-prompt review (noise-level, all arms, both families) versus the multi-seed deep finder with a verifier behind it (0.571 findable recall, the only local configuration with signal).

One more check closed the program. We re-ran the three arms against the review-time pinned diffs, where 14 of the 15 ground-truth items are findable instead of 7. The deep finder caught 8 of 14, the same 0.571, and this time both of the correctness-critical items, which had been ghosts at the final head. Framing alone caught 4 of 14 with no criticals; the fast model's card configuration caught nothing on any basis. (Re-judged with Sonnet 5 before publication: deep 9 of 14, framing 1 to 2 of 14, criticals and ranking unchanged; the appendix has the per-item reasons.)

Interlude: build vs. adopt, measured on the same instrument

Mid-program we tried to replace our home-built gate with alibaba/open-code-review, a polished precision-first agent, and separately tried distilling its per-language rule docs into our finder prompt as a checklist. Both tests ran on the pre-audit instrument, so the absolute numbers carry that era's caveat, but both decisions survive it. The agent's precision bet is real and its context appetite is not local-servable (~471K input tokens to review one file); its recall came in at a fifth of ours on the same judge, a gap no re-audit direction can flip. The checklist made our finder worse on every axis and destabilized it into spray blowouts and empty collapses. Borrowed wisdom carries its design philosophy with it, and each verdict cost about a day because the instrument already existed.

What runs today, and the honest ceiling

The shipped gate: semgrep floor on every push, deep multi-seed finder for significant pushes, verifier on top, frontier reviewer after. The deep pass runs in about six minutes on the box, at $0 marginal. On the audited basis it catches a bit over half of the findable known bugs, with spray-level per-run precision (this feeds a verifier and a human, not a dashboard). The deterministic layer stays the only tier with veto power, because the local verifier over-refutes: it has talked itself out of real block-severity bugs.

The local tier does not replace frontier review, and the audited numbers say it plainer than the optimistic ones ever could. What it does is cheap, early, and on-premises triage, with a measured boundary instead of a vibe.

The playbook

Verify your instrument before your subject. Guided decoding silently disabled thinking and we nearly published "reasoning helps." Check raw completions, not scores alone; probe one completion by hand before batching any new model mode.
Audit your judge, then audit your ground truth. Our first judge inflated recall 10×; our ground truth was 73% ghosts. Both audits together cost under two days and less than $5, and they were worth more than every sweep combined. An adversarial confirmation pass (the judge must articulate the shared failure mode, decisions persisted) is cheap to build and repeatable.
Pin ground truth to the code state its evidence saw. Review comments are made against intermediate commits; replaying final diffs manufactures ghosts. If an item's location isn't in the pinned diff, it isn't findable, and it doesn't belong in the denominator.
Distrust results that flatter your setup, in proportion to the flattery. Every instrument error in this program pushed the same direction: toward the conclusion we wanted. Ghosts inflate recall, lenient judges inflate recall, theme-matching inflates recall. Honest instruments mostly make your numbers worse.
Strict matchers rank, judges calibrate, and neither is trustworthy unverified. The strict scorer's celebrated ranking also failed to replicate. Treat small strict-basis deltas as noise, and never make an adopt/reject call on an unaudited instrument.
Keep the golden set alive, and audit it on a schedule. Every decision in this program cost about a day and single-digit dollars because the harness already existed. But the harness is only an asset while its ground truth is honest; ours went a full month before anyone checked.

The gate this produced reviews every significant push on my repos, on my own hardware, for free. It catches about half of what's actually there, we know that number is real because we tried very hard to kill it, and that combination beats every prettier number this program ever produced.

This is Spark Bench: what one DGX Spark on a desk can and cannot be trusted to do, measured honestly. Next: the gate above runs on two local models that share one 128 GB box with everything else my coding agents need, and the config that makes that fit is the next article. The 47 bugs that actually shipped, and how the same gate scores against them, follow later this week.

Appendix: the numbers behind the story

The three measurements that changed the verdict, side by side. Union recall means "caught by at least one of the arm's runs"; the findable basis counts only ground-truth items that exist in the diff the reviewer actually saw. All three columns score the same seven pull requests (an eighth, PR 144, exceeds the models' context window on every basis and is excluded from every arm). Two of the seven carry no ground-truth items and count only toward precision; the 14 findable items sit on the other five. Qwen3.8-27B NVFP4 and Nemotron-3.5-Lightning-30B-A3B NVFP4, vLLM 0.27.1, 2026-08 to 2026-09.

arm	lenient judge, final-head diffs (Aug 28)	confirmed judge, final-head diffs (Sep 13)	confirmed judge, pinned diffs (Sep 14)
single prompt, framing (q38)	0.667	0.143 findable (1 of 7)	0.286 findable (4 of 14)
single prompt, card sampling (Nemotron)	0.400	0.0	0.0
deep: 3 seeds + verifier (production)	reported as 1.0	0.571 findable (4 of 7)	0.571 findable (8 of 14), both criticals

Judge vintage check (Sep 14, Sonnet 5)

The judge of record for this article is Sonnet 4.6, because the audit in Act 3 is an audit of that judge's decisions. Sonnet 5 shipped before publication, so the pinned-basis column was re-judged with it on identical inputs: same finder output, same ground truth, same mapper and confirmation prompts. Claude 5 models refuse a fixed temperature, so the re-judge ran twice.

arm, pinned diffs	Sonnet 4.6	Sonnet 5, pass 1	Sonnet 5, pass 2
single prompt, framing (q38)	4 of 14	2 of 14	1 of 14
single prompt, card sampling (Nemotron)	0	0	0
deep: 3 seeds + verifier	8 of 14, 2 of 2 criticals	9 of 14, 2 of 2	9 of 14, 2 of 2

Every disagreement was read by hand. The deep arm's extra item is a PR 42 finding that names the same missing CLI validation as the ground-truth item with a differently worded consequence; 4.6's verifier rejected it on the wording, Sonnet 5 confirmed it, and I agree with Sonnet 5. The framing arm's lost items all come from its tests angle, which states a defect in the form "this path has no guard and no test covers it"; Sonnet 5 reads those as coverage findings, which the rubric excludes, and dropped one more of them on its second pass than its first. Whether "no guard, add a test" counts as finding the bug is a rubric call, and the honest statement is that the framing arm found between 1 and 4 of 14 depending on it. What does not move under either judge: the ranking (deep, then framing, then card sampling), zero criticals for the single-prompt arms, and both criticals for the deep arm. The judge default in the harness is Sonnet 5 from this date forward, and every scorecard now records which model judged it.

What pinning did to the ground truth itself:

basis	located items	findable	ghosts
final-head diffs	99	26	73
review-time pinned diffs	99	91	8

The confirmed-judge protocol, in four sentences. A mapper call proposes finding-to-ground-truth matches over the full text of both sides (the first judge saw 280 characters of each, which is how it missed self-retractions). A second, adversarial call then takes each proposed pair alone and must articulate the shared failure mode before it may confirm; its default is no. Only confirmed pairs count, and every decision is persisted with its rationale so the audit can be repeated. Repeated twice on the same inputs, it produced the same matches both times.

What is not published: the pinned diffs and the raw review outputs, because both are the source code of the repositories under review. The specs, the harness, and every aggregate above are the reproducible part.