Spark Bench · Article 6
The Review Bot Read 28 of My PRs and Caught 3 of 35 Bugs That Shipped
I pay CodeRabbit $30 a month and run a free local reviewer on the DGX Spark on my desk, so I scored both against 47 bugs that shipped from six of my repos, curated from 5,215 fix pairs found with git blame. CodeRabbit flagged 3 of the 35 on the 28 PRs it reviewed. Qwen3.8-27B (NVFP4, vLLM 0.27.1), three seeds plus a verifier, finds 19 of 40 in 108 findings per PR; one prompt finds 6, and Nemotron-3.5-Lightning-30B-A3B (NVFP4) finds 3. A Claude Sonnet 5 triage keeps 12 of the 19 in 13 per PR. Run the local finder plus a Sonnet 5 triage before every push, at $0.48 a PR, and the paid bot once, on the release PR.
The stack
One NVIDIA DGX Spark, GB10, 128 GB unified memory. Finder: Qwen3.8-27B NVFP4 under vLLM 0.27.1, three framed seeds plus a verifier, 58K-token input budget. Triage and judge: Claude Sonnet 5 over the API, every verdict on disk. CodeRabbit stays paid, for one pass per release pull request.
Act 1: Ground truth comes from fix commits
My earlier review benchmark scored against reviewer comments, and 73% of that ground
truth turned out to be ghosts. A bug that shipped leaves better evidence: a fix
commit, and git blame on the lines it touched names the merged pull request that
put them there. My Claude Code sessions and I walked every fix-shaped commit in six
of my repos, called A through F here, and 5,215 fix pairs came out.
Deterministic filters dropped merge commits and fixes that named a review round, then ranked the rest by how concentrated each fix was on one origin PR. 137 candidates on 110 pull requests survived. Six Sonnet curators, read-only, default answer drop, kept 43. I audited every keep against its commit and overrode nine; four keeps were not bugs at merge time.
The dominant drop reason was history. Repo B went multi-tenant on 2026-03-29, and most later “scope to tenant” fixes corrected code that was right for the schema it was written against. A fix to code that was right at merge is not a shipped bug.
The corpus is 47 incidents on 37 pull requests, 8 security and 39 correctness, each pinned to an added line of the diff. Not one came from a reviewer comment. Six of those PRs exceed my finder’s input window, which leaves 40 incidents on 31 PRs it can read. The scoring still runs on my own harness, not the open-source one the next article adopts, so every raw output behind these numbers is pinned in the Appendix.
Act 2: The paid bot flagged 3, and all three merged
I gave every comment CodeRabbit left on those pull requests, including the outside-diff ones it hides in a collapsed block, to the judge that scores my local arms, with the same adversarial confirmation step.
| CodeRabbit on the 28 golden PRs it reviewed | |
|---|---|
| comments, all kinds | 1,034 |
| comments it labelled actionable / nitpick | 638 / 176 |
| shipped bugs on those PRs | 35 |
| shipped bugs it flagged | 3 |
| precision against shipped bugs, on actionable | 0.009 |
| review threads resolved or outdated before merge | 329 of 564 |
The three it flagged were real. On repo B #454 it labelled a migration Critical, said the skipped backfill would make the NOT NULL step fail on any existing database, and the PR merged; the migration failed on Railway two days later exactly as described. On repo E #84 it named the one-level regex that could not parse nested JSON. On repo B #395 it asked for the selects to be wired through the form controller, which is the bug that later shipped.
The last row is the bot’s defense, and it is a real one. Fifty-eight percent of its threads were resolved or went stale before merge, which means someone changed the code under them. A bug fixed that way never becomes an incident, so a shipped-bug basis cannot see it. This basis counts what CodeRabbit missed among the bugs that got through, not what it prevented.
Act 3: The local gate finds 19, and one prompt finds 6
My production gate runs Qwen3.8 on three seeds with a framing prompt, unions the findings, and passes them through a verifier. Over the readable PRs it caught 19 incidents, including 3 of the 4 criticals, and security items at the same rate as correctness. I read all 19 confirmations. Each names the concrete defect: a tenant predicate missing from one query, a verification token stored unhashed, a check-then-act race between two webhook deliveries, a dict double-encoded into a JSONB string.
| arm, same 40 incidents | union recall | 95% CI | precision per run | findings |
|---|---|---|---|---|
| Qwen3.8, framed, 3 seeds + verifier | 19 of 40, 0.475 | 0.31 to 0.67 | 0.019 | 3,355 |
| Qwen3.8, one prompt, 4 angles, 3 seeds | 6 of 40, 0.150 | 0.05 to 0.27 | 0.023 | 522 |
| Nemotron-3.5-Lightning, one prompt, 4 angles, 3 seeds | 3 of 40, 0.075 | 0.00 to 0.16 | 0.005 | 564 |
The first two rows are the same checkpoint. The framing prompt and the verified three-seed union are worth 3.2 times the recall, and all six single-prompt catches sit inside the nineteen.
The price is the findings column: 64 of the gate’s 3,355 matched a shipped bug. The local verifier is the known weak link: it over-refutes. The question was whether anything cheap could turn that spray into a list a person reads.
Act 4: A frontier triage keeps 12 in a list of 13
I sent every finding to Sonnet 5 in batches of fifteen, sorted by file, with only the diff hunks those files touched. Each finding got a verdict: actionable, with the diff line that grounds it; possible, when the diff alone could not decide; or drop, with a refuting line or the reason it is not a defect. A correctness or security claim is never dropped without a refuting line.
| what the reader sees, per PR | findings | shipped bugs kept, of 19 |
|---|---|---|
| the finder’s raw output | 108 | 19 |
| actionable only | 13.5 | 12 |
| actionable plus possible | 30 | 17 to 19 |
| actionable, block and major severity only | 4.5 | 6 |
Three full passes, $15 each. The actionable count held at 12 in every pass; the combined tier ranged from 17 to 19. Severity was the dial I expected to use, and it is the wrong one: shipped bugs mostly read as minor to the triage. Verdict is the dial.
The incidents that landed in “possible” all had the same shape. The diff showed the suspicious line, and the fact that decided it lived one definition away, such as the asyncpg codec in a pool module or the tenant model behind a schema. On 2026-09-22 I tried fetching it: a second pass grepped each named symbol and asked again. It moved 2 of those incidents to actionable, dropped 3 real ones because grep found nothing, and promoted 96 non-bugs. A missing grep hit is not a refuting line, and the model dropped anyway. Not adopted; retrieving the definition itself, one block, is queued.
The decision
Since 2026-09-20 the pre-push pass on my repos is the local finder plus a Sonnet 5 triage, at $0.48 a PR, and CodeRabbit sees one diff per release pull request. The constraint is reading time: 13 findings with a cited line each is a list I read, and the raw spray is not. At my volume of 191 merged PRs a month the triage is under $100.
The alternatives lost on the same basis. CodeRabbit on every push found 2 shipped bugs on the 22 PRs it and the local finder both covered, and every bug it caught there, another arm caught too. Its hourly review allowance shrinks with the past week’s use, so one round on a final diff never drains it. One prompt to the same model lost in Act 3. Alibaba’s open-code-review, an agentic harness, found 13 of 40 driving Sonnet 5 at $2.04 a PR, about $390 a month at my volume. It also reads the diffs my finder cannot, which makes it a release-PR candidate (Appendix).
The cost is CodeRabbit’s column. It reviewed four of the six pull requests whose diffs exceed my finder’s window, and one of its three catches sits on a PR the finder never read. Its seat is now a second model family on the final diff, and whether it keeps that seat or open-code-review on Sonnet 5 takes it is queued. The triage has misses of its own. It is not local: on the day I ran the passes, the API account ran dry mid-sweep. Reading only the actionable tier drops 7 incidents the finder had. With Sonnet 5 inside it, open-code-review found 5 bugs on those PRs that the finder never did, so the triage is the cheap floor, not the ceiling.
The misses cluster on the biggest diffs, so I tried the obvious lever first: doubling the per-seed ceiling to 3,600 seconds on the four PRs that timed out twice. It caught 2 of their 5 items instead of 1, for about ten GPU hours, at a precision of 0.008. The ceiling stays at 1,800 seconds. Queued instead: more seeds, since the union of the twelve seeds I have run across arms reaches 26 of 40; then reviewing a large diff file by file rather than whole; definition retrieval for the possible tier; open-code-review on release PRs in place of a second CodeRabbit round; and a re-score whenever a new incident lands in the golden set.
The playbook
- Mine git history for ground truth before you score any reviewer against it.
- Date the schema: a fix to code that was right when it merged is not a shipped bug.
- Score your paid reviewer against the bugs that shipped, and count what it prevented separately.
- Recall is cheap locally and reading is the cost, so buy the filter, not the finder.
- Rank by verdict, not severity.
- Never let a triage drop a correctness claim without a refuting line.
- Run the rate-limited bot once, on the final diff.
This is Spark Bench: what one DGX Spark on a desk can and cannot be trusted to do, measured honestly. Next: My LLM Eval Harness Failed More Often Than the Models It Graded, an audit of the harness that picks my agent’s driver model, run the way I audit the models. Follow for the tally.
Appendix: the numbers behind the story
Everything here is a view over two findings documents and the scoring files they name.
Golden set: 47 incidents on 37 pull requests from six of my repos, here repo A through
F; 40 on the 31 PRs within the local finder’s 58K-token window. Local gate: Qwen3.8-27B
NVFP4 on vLLM 0.27.1, three seeds, the production --deep framing plus verifier,
1,800-second per-seed ceiling, run 2026-09-14 to 2026-09-15. Judge for every arm on the
incident basis: Claude Sonnet 5, mapper plus adversarial confirmation, every decision
on disk, 2026-09-15 to 2026-09-21. The seeded basis has no judge.
The local gate’s run
| incident basis | seeded basis | |
|---|---|---|
| pull requests | 37 | 12 |
| cells (PRs × seeds) | 111 | 36 |
| cells with findings | 88 | 33 |
| over-context PRs (excluded) | 6, carrying 7 items | 1, carrying 1 item |
| chronic timeouts (1,800 s twice) | 5 cells on 4 PRs, each PR still has a seed with findings | 0 |
| items in the denominator | 40 on 31 PRs | 11 on 11 PRs |
| findings emitted | 3,355 | 862 |
| judge calls | 160 (87 mapper, 73 confirmation), 0 failed records | none |
Incident basis, item by item
Caught (19). repo E #84 regex JSON parse fails on nested or fenced
output; #84 cache treats only pass as done; #122 popover missing
bottom-of-viewport clamp. repo B #31 non-atomic findUnique-then-create
race; #31 asset_ready before asset_created loses playback data; #265 admin
reports unscoped by tenant; #266 verification token stored unhashed; #389
partial unique index missing tenantId; #454 migration runs SET NOT NULL
after skipping the backfill. repo F #11 hardcoded demo admin password; #18
auth falls back to POC_USER_ID. repo C #23 truthy check coerces explicit 0
to NULL; #26 prior_ytd left cumulative while current_ytd rebased; #164
expenses CTE amount < 0 drops refunds; #294 minority filter sums all trusts;
#321 total from unquantized decimals mismatches quantized components. repo A #39
sync loop lacks per-service exception handling; #46 and #149 jsonb
double-encode.
Missed (21, by curation id). C026 repo B #265 (security, critical: cross-tenant report access; its sibling item on the same PR was caught), C044 repo F #11, C129 repo E #73, C079 repo D #7 (security); C071 repo C #491, C072 #20, C073 #5, C056 #294, C054 #68, C065 #69; C043 and C045 repo F #88; C096 repo A #51, C103 #139, C100 #236, C090 (two items) and C109 (two items) repo A #75; C037 repo B #267; C118 repo E #114.
Outside the denominator (7 items on 6 PRs). repo E #49 and #117, repo B #395, repo C #271, repo D #51, repo A #5: the pinned diff exceeds the finder’s input budget on every seed (the largest is about 309K tokens).
| incident basis, by slice | caught / total |
|---|---|
| security | 4 / 8 |
| correctness | 15 / 32 |
| critical / major / minor | 3/4, 12/29, 4/7 |
| repo B | 6 / 8 |
| repo C | 5 / 11 |
| repo E | 3 / 5 |
| repo A | 3 / 10 |
| repo F | 2 / 5 |
| repo D | 0 / 1 |
| per seed | 15, 14, 11 of 40 |
| bootstrap 95% CI on union recall | 0.31 to 0.67 |
Every arm on the same 31 PRs and 40 incidents
| arm | shipped bugs caught | findings | precision |
|---|---|---|---|
| Qwen3.8, three framed seeds plus verifier (the local gate) | 19 | 3,355 | 0.019 |
| the gate plus Sonnet 5 triage, actionable | 12 | 419 | |
| the gate plus Sonnet 5 triage, actionable and possible | 17 to 19 | 876 to 957 | |
| open-code-review 1.12.7 driving Qwen3.8 | 7 | 261 | 0.031 |
| open-code-review 1.12.7 driving Sonnet 5 | 13 (16 of 47 on all 37 PRs; 5 not found by any local seed) | 357 | 0.045 |
| Qwen3.8, one prompt, four angles, three seeds | 6 | 522 | 0.023 |
| Nemotron-3.5-Lightning-30B-A3B, one prompt, four angles, three seeds | 3 | 564 | 0.005 |
| CodeRabbit, on the 22 of these PRs it reviewed, 28 incidents | 2 | 623 | 0.005 |
Single-prompt arms (added 2026-09-18)
Same 31 PRs and 40-item denominator as the deep arm, four angles (correctness, tests, quality, security) times three seeds per arm, judge Claude Sonnet 5 with zero judge failures. The Qwen3.8 arm ran serially on 2026-09-16, then under a 4-worker concurrent runner on 2026-09-17 after a measured neighbour effect (a 4-worker Nemotron job on the other engine had halved its decode); every timeout produced under concurrency, 25 cells, was quarantined and re-run one at a time at full speed on 2026-09-18, 14 recovered and 11 timed out again and stand as chronic. The Nemotron arm ran at 4 workers on its own engine on 2026-09-17, 372 cells in 74 minutes.
| arm | records | findings | union recall | 95% CI | per-run recall | precision / run | shipped bugs caught |
|---|---|---|---|---|---|---|---|
| Qwen3.8, thinking, one prompt | 355 of 372 (17 chronic timeouts) | 522 | 6 of 40, 0.150 | 0.05 to 0.27 | 0.100 | 0.023 | B #454, repo F #11, #18, repo A #46, #75, #149 |
| Nemotron-3.5-Lightning, card sampling, one prompt | 368 of 372 | 564 | 3 of 40, 0.075 | 0.00 to 0.16 | 0.033 | 0.005 | B #31, #454, repo F #88 |
The two single-prompt arms share one catch (repo B #454). The Qwen3.8 single-prompt arm catches nothing the deep arm missed. The chronic timeouts are all on the correctness angle over the largest diffs; the deep arm’s own five double-timeout cells are on the same PRs.
CodeRabbit, per pull request
28 PRs reviewed of 37; the nine it did not review are B #31, #265, #266, #267, F #18, C #23, #26, #294, #321. Threads are GitHub review threads started by the bot; “acted on” is resolved or outdated before merge.
| PR | comments | threads acted on / threads | shipped bugs on PR | flagged |
|---|---|---|---|---|
| E #49 | 110 | 24 / 55 | 1 | 0 |
| E #73 | 41 | 10 / 20 | 1 | 0 |
| E #84 | 36 | 8 / 20 | 2 | 1 |
| E #114 | 33 | 6 / 16 | 1 | 0 |
| E #117 | 22 | 3 / 13 | 1 | 0 |
| E #122 | 19 | 6 / 10 | 1 | 0 |
| B #389 | 20 | 1 / 7 | 1 | 0 |
| B #395 | 151 | 21 / 68 | 1 | 1 |
| B #454 | 62 | 41 / 49 | 1 | 1 |
| F #11 | 6 | 3 / 3 | 2 | 0 |
| F #88 | 76 | 48 / 53 | 2 | 0 |
| C #5 | 48 | 18 / 24 | 1 | 0 |
| C #20 | 20 | 8 / 9 | 1 | 0 |
| C #68 | 9 | 5 / 5 | 1 | 0 |
| C #69 | 2 | 1 / 1 | 1 | 0 |
| C #164 | 23 | 12 / 12 | 1 | 0 |
| C #271 | 28 | 1 / 14 | 1 | 0 |
| C #491 | 18 | 8 / 9 | 1 | 0 |
| D #7 | 49 | 23 / 24 | 1 | 0 |
| D #51 | 58 | 1 / 34 | 1 | 0 |
| A #5 | 42 | 30 / 42 | 1 | 0 |
| A #39 | 43 | 15 / 24 | 1 | 0 |
| A #46 | 23 | 6 / 9 | 1 | 0 |
| A #51 | 6 | 2 / 2 | 1 | 0 |
| A #75 | 37 | 8 / 15 | 1 | 0 |
| A #139 | 5 | 2 / 2 | 1 | 0 |
| A #149 | 14 | 5 / 6 | 1 | 0 |
| A #236 | 33 | 13 / 18 | 1 | 0 |
Its three catches, confirmed by reading the comment against the incident: repo E #84, a JSON-extraction regex limited to one level of nesting (comment at line 449, incident at 455); repo B #395, three select inputs bound with a one-off setValue instead of the form controller, so a reset left them stale (line 231 and 217); repo B #454, a migration that skipped its backfill when the default tenant was missing and then added NOT NULL constraints (line 18 and 16, labelled Critical, fixed in production on 2026-05-01).
The triage, three passes
Batches of 15 findings sorted by file; only the diff hunks of the referenced files, capped at 120K characters per batch; Claude Sonnet 5, 16,000-token output cap; verdict actionable, possible, or drop with a cited line. True catches are the 64 findings the judge confirmed against the 19 incidents.
| pass | dedupe before the model | findings judged | actionable per PR, incidents | actionable plus possible per PR, incidents | cost |
|---|---|---|---|---|---|
| 1 | none | 3,355 | 13.5, 12 | 30.9, 19 | $15.01 |
| 2 | ±5 lines, ≥0.5 overlap | 3,322 | 13.5, 11* | 29.9, 19* | $14.99 |
| 3 | ±10 lines, ≥0.3 overlap | 2,968 | 13.3, 12 | 28.3, 17 | $13.77 |
* Pass 2’s per-finding file was overwritten by pass 3 through a unit-file quoting mistake; its incident counts are from the run summary on the judge’s first-pass basis and are approximate. Verdicts over pass 1: 419 actionable (11 block, 128 major, 280 minor), 538 possible, 2,398 drop, of which 659 were labelled duplicates.
One pass lost a real incident to my own dedupe. Merging near-duplicate findings before the model collapsed three phrasings of the same bug and kept the weakest, which the triage dropped as a style preference. The undeduped pass had kept the sharpest phrasing as actionable. Semantic dedupe is the model’s job; mine now merges only near-verbatim repeats.
Pass 3’s two losses: repo C #26, where the model read a test docstring as proof the behaviour was by design, a wrong refutation; and repo B #266, the dedupe collapse described above.
The nine “possible” incidents and the fact outside the diff that would decide each: repo E #84, the real shape of the validation payload; repo B #265, the tenant model behind the admin reports route; repo B #389, whether a unique index on scheduled slots needs a tenant column; repo C #26, how the rebased equity column is rendered; repo C #69, whether re-ingesting archived intake documents is expected; repo C #164, whether the old profit-and-loss view filtered by sign; repo C #294, two items, the trust-versus-source-trust semantics and a display-name collision; repo A #149, whether the connection pool auto-encodes JSONB.
Dedupe calibration against the 642 same-file pairs the model labelled duplicates: median distance 4 lines, median token overlap 0.26; a ±5-line, 0.5-overlap rule catches 2 percent of them, ±10 and 0.3 catches 29 percent with no true catch merged into a non-catch, ±30 and 0.2 catches 71 percent and merges 52 true catches away.
The 3,600-second ceiling run
The four PRs whose deep-arm cells hit the 1,800-second ceiling twice (repo F #88, repo C #164 and #491, repo A #236; 5 items) were re-run on 2026-09-19 at 3,600 seconds per seed, two workers on Qwen3.8, same judge.
| at 1,800 s (same 4 PRs) | at 3,600 s | |
|---|---|---|
| items caught | 1 of 5 (repo C #164) | 2 of 5 |
| cells with findings | 7 of 12 | 7 of 12 |
| cells failed | 5 timeouts | 5 output-truncation errors (32,768-token cap, 52 to 57 min each) |
| longest successful cell | the 1,800 s ceiling | repo F #88 seed 2: 155 findings in 158 min |
| precision per run | 0.019 | 0.008 |
Given twice the time, the finder on the largest diffs either runs into its 32K output cap and returns nothing or sprays. One extra item cost about ten GPU hours. The binding constraints on big diffs are output budget and diff size, not wall clock, which is why per-file review is queued and the ceiling is not.
Planted bugs: the seeded basis
The second basis is synthetic: 15 bug classes distilled from the incidents, each reintroduced by a model into a recent merged PR outside every golden set, with a verifier deciding whether the edit counts. Twelve came out; eleven fit the window. Strict location scoring, a finding within five lines of the planted line with the matching category and no judge, gave 10 of 11. Read by hand, all 11: the one strict miss was named by all three seeds twenty lines away, at the line where the SQL string executes rather than where it is defined. That is saturation. These are the classes the finder was tuned against, each a single local edit with a comment left contradicting it, so a gate that catches 11 of 11 planted bugs and 19 of 40 shipped ones is telling you to trust the 19. The seeded basis was not run for the single-prompt arms.
Strict = a finding in the seeded file within five lines of the seeded line with the seeded category. Audited = a hand read of every in-window finding and every same-file finding on a miss, asking only whether the finding names the planted defect. repo B #1063 (tenant scope omitted in a migration script) is over-context on all three seeds and excluded.
| PR | class | category / severity | seeded location | strict: seeds caught | audited: seeds caught |
|---|---|---|---|---|---|
| E #171 | raw-error-to-user | security / minor | apps/reader/src/app/api/session/route.ts:51 |
2, 3 | 1, 2, 3 |
| E #173 | validator-skips-sibling-field | correctness / major | services/annotator/annotator/tier_parser.py:41 |
1, 2, 3 | 1, 2, 3 |
| E #176 | auth-check-missing | security / major | apps/reader/src/app/api/progress/stats/route.ts:14 |
1, 2, 3 | 1, 2, 3 |
| B #1057 | tenant-scope-omitted | security / critical | apps/web/src/lib/admin/site-analytics-queries.ts:138 |
2, 3 | 1, 2, 3 |
| F #130 | falsy-zero-coerced | correctness / major | web/src/app/api/finance/leaderboard/route.ts:121 |
1, 2 | 1, 2, 3 |
| F #135 | falsy-zero-coerced | correctness / major | web/src/app/api/finance/leaderboard/summary.ts:5 |
1, 2, 3 | 1, 2, 3 |
| C #576 | validator-skips-sibling-field | correctness / major | scripts/seed_governance_records.py:122 |
1, 2 | 1, 2, 3 |
| C #578 | except-too-narrow | correctness / major | scripts/services/governance/backfill.py:155 |
1, 2, 3 | 1, 2, 3 |
| C #585 | validator-skips-sibling-field | correctness / major | api/models/governance.py:110 |
2, 3 | 3 |
| C #586 | dedupe-key-too-coarse | correctness / major | scripts/services/employment_seed/seed.py:76 |
none | 1, 2, 3 |
| A #414 | unit-scale-mismatch | correctness / major | review/finder.py:34 |
1, 2, 3 | 1, 2, 3 |
Where strict and audited disagree: #586 is named by every seed at lines 96 to
103, where the SQL string is executed, not at line 76 where it is defined.
#171 seed 1 (line 44), #130 seed 3 (line 115), and #576 seed 3 (line 131) name
the defect six to nine lines outside the window. #1057 seed 1 names the missing
tenant predicate at line 131 but files it as correctness, and the seed is
security. #585 seed 2’s two in-window findings are about signer_name null
handling and Pydantic in-place mutation, not the planted bug; seed 3 names the
missing signer_role strip at line 111 as a quality nit, which the category
rule ignores, and gets its credit from an unrelated finding at line 109
instead. Two more cells (#173 seed 1, #576 seed 2) get the right verdict from
the wrong finding while the real catch sits eight and seventeen lines away.
| seeded basis | strict | audited |
|---|---|---|
| union recall | 10 of 11 | 11 of 11 |
| per seed | 7, 10, 8 | 10, 10, 11 |
| cells agreeing with the hand read | 25 of 33 | |
| bootstrap 95% CI on strict union recall | 0.73 to 1.0 |
open-code-review
Alibaba’s open-code-review is the agentic tool people told me to try: it groups the files, plans, lets an agent search the repository, then filters. I ran version 1.12.7 over the same 37 PRs twice, once with Qwen3.8 as its model and once with Sonnet 5.
| open-code-review on the 31 PRs the local finder covers | shipped bugs, of 40 | findings | precision |
|---|---|---|---|
| driving Qwen3.8, 78M input tokens, 5 to 50 min per PR | 7 | 261 | 0.031 |
| driving Sonnet 5, $2.04 per PR with caching, 8 to 16 min per PR | 13 | 357 | 0.045 |
| for comparison: Qwen3.8, one prompt, no harness | 6 | 522 | 0.023 |
| for comparison: my finder plus triage, actionable | 12 | 419 |
On the local model, the harness is one catch better than a single prompt to the same model, and it is the most precise local arm. It read the whole repository to get there, at 806 tool calls and up to two million input tokens on one PR, and it found less than half of what three framed seeds found reading only the diff. With Sonnet 5 inside it the same harness is a different tool: 13 of 40 at the best precision of any arm, and 5 of those 13 are bugs the local finder never produced in any seed. It also read the six pull requests too large for the local finder and found 3 more there, 16 of 47 overall. The union of the local finder and the Sonnet harness is 24 of 40. They are complementary, and the harness costs four times the triage per PR.
Version 1.12.7 (2026-09-19 release, linux-arm64). Each golden PR replayed at its recorded base and head commits in a detached worktree; the tool’s own excludes plus the golden set’s; four concurrent subtasks per PR, 30-minute per-task timeout, review effort medium, its post-filter on. Qwen3.8 arm: thinking off via the chat template, 78M input tokens and 0.82M output over 37 PRs, 517 comments, 2026-09-19 23:21 to 2026-09-20 22:53, on the same engine as the local gate. Sonnet 5 arm: 597 comments over 37 PRs, $75.46 at list price with prompt caching ($2.04 per PR; 8 to 16 minutes on the largest diffs), 2026-09-20 15:45 to 2026-09-21 12:32; 8 PRs were re-run after the API balance reached zero mid-sweep on 2026-09-20 at 18:06, which the tool reported as failed rather than empty. Judge: 0 failed records on both arms. Its 5 catches the local finder never produced: repo C #69, repo A #51, two on repo A #75, and repo A #236; its 3 on the over-window PRs: repo B #395 and two on repo A #5.
Costs, this article
Sonnet 5 at $2 per million input tokens and $10 per million output, cache reads at $0.20. Three triage passes $43.77; CodeRabbit adjudication under $1; open-code-review with Sonnet 5 $75.46; judge calls for the two open-code-review arms about $2. CodeRabbit’s list price is $30 a month for one seat; its per-file overage when rate-limited is quoted from memory and unmeasured here.
Files
docs/superpowers/specs/2026-09-15-golden-v3-scoring-findings.md (the findings
document, complete), adjudicated_v3_incidents_deep_2026-09-15.json and
judge_matches_v3_incidents_deep_2026-09-15.json (incident basis, deep arm, every
judge decision), scored_v3_seeded_deep_2026-09-15.json (seeded basis, every cell),
adjudicated_v3_single_2026-09-18.json and judge_matches_v3_single_2026-09-18.json
(single-prompt arms), all under data/qwen_reviewer_benchmark/ in the lab repo
alongside the ground-truth directories named at the top of this article. The raw
finder output and the pinned diffs quote application code and are not published.
Raw outputs (pinned)
The raw files behind this article stay outside the published page; each is pinned
by SHA-256 below so any of the numbers above can be re-audited against the exact
bytes that produced it. File paths show each repository’s letter in place of its name (repo-B__1057 stands for that repository’s file); the hash is of the real file.
| file | what it backs | sha256 |
|---|---|---|
adjudicated_v3_incidents_deep_2026-09-15.json |
incident-basis deep-arm scoring and the reused local-gate baseline row: union recall 19 of 40, 3 of 4 criticals, 3,355 findings, 0.019 precision, the per-slice and per-repo caught/total counts | 2640577398701a165d0b833329920e291e53df24982aab47b0c88400bdba2ba8 |
judge_matches_v3_incidents_deep_2026-09-15.json |
incident-basis judge decisions: 160 judge calls (87 mapper, 73 confirmation), 0 failed records, the caught (19) and missed (21) item-by-item lists; the per-finding decisions the triage passes re-score against, 73 findings / 23 incidents on the mapper basis, 64 findings / 19 incidents confirmed | 428ebf67d45aa0636223d8d3996f14ec3bf5b8ce495c5b641e52655bdf231afb |
adjudicated_v3_single_2026-09-18.json |
single-prompt arms summary: Qwen3.8, one prompt, four angles, three seeds, 6 of 40, 522 findings, 0.023 precision; Nemotron 3 of 40; records/findings/precision per arm | 9bea3ab2ac0b3d01ad7d7433dccca80075d66a2994b351ed68f92f0cf5a9b793 |
judge_matches_v3_single_2026-09-18.json |
single-prompt arms per-finding judge decisions, the shipped-bugs-caught lists, 0 judge failures | e2da945672a0312dc00c4c486f15d25bd31f0c1870e1aee115507863c0cff824 |
adjudicated_v3_coderabbit_2026-09-20.json |
CodeRabbit summary: 35 shipped bugs on the PRs it reviewed, 3 flagged, recall 0.086, precision 0.006 | a03f289579b0b07a0d9ccc6d4b6681abb6365e257f66124fc4912d3984812677 |
judge_matches_v3_coderabbit_2026-09-20.json |
CodeRabbit per-PR comment counts (n_findings, the “comments” column of the per-PR table) and per-PR matched-incident decisions for all 28 reviewed PRs |
f0c0870662985dbbb584d9d63259734009e5f8dff8ac8db320ccf04c7a222d05 |
adjudicated_v3_ocr_q38_2026-09-20.json |
open-code-review driving Qwen3.8, on the 31-PR comparison: 7 of 40, recall 0.175, precision 0.031 | 902491c3177b8c5dc6e5f4f1ca2284464083cace52de1bcc6a9806554713d941 |
judge_matches_v3_ocr_q38_2026-09-20.json |
the same arm’s 261 findings across 31 PRs and the matched-incident detail behind the 7-of-40 and 0.031 figures | b6282e586f6731dabe862440414361242247dd9656ad0dba7e2c17a77662e125 |
adjudicated_v3_ocr_sonnet5_2026-09-21.json |
open-code-review driving Sonnet 5, all-37-PR basis: 16 of 47, recall 0.34 | 66e54885ebb79f71d8e10f1713bdd694186a575f4da2d1aa1415b531527041e8 |
judge_matches_v3_ocr_sonnet5_2026-09-21.json |
the same arm’s per-PR findings and matches, from which the 31-PR comparison figures (13 of 40, 357 findings, 0.045 precision) are re-scored, and the 5 catches the local finder never produced | c3c82c260b20687e6a1717ccb7ac5270cfca6ca68681acab6dfb90d8dab6589c |
triage_v3_2026-09-20.json |
triage pass 1 (no dedupe): 3,355 findings judged, 419 actionable (13.5/PR), 957 actionable+possible (30.9/PR), the duplicate-label pairs behind the dedupe-calibration numbers | bfce09084d9c4d0026a789acaea6c5f57a1321fb8db341385b28dfe46652e5c2 |
triage_v3_dedupe_run2_2026-09-20.json |
triage pass 3 (±10 lines, ≥0.3 overlap dedupe): 2,968 findings judged, 13.3/PR actionable (12 incidents), 28.3/PR actionable+possible (17 incidents), cost $13.77 | 4efe3c989311f0d3570bca3b135ba6d6d0265e9e105686d7c0f3598515e4cb85 |
results_ocr_v3_2026-09-19/ (74 files, not tracked in git; manifest hash) |
the Qwen3.8 open-code-review arm’s raw per-PR runs: per-PR elapsed time (5 to 51 minutes on the 31 comparison PRs), open-code-review v1.12.7 version stamp, coverage. Manifest hash: each of the 74 files was hashed with sha256sum, the 74 lines sorted by path, and that combined text hashed again with sha256sum |
95b59917b3679c1ba917b1654504c00a9e27d6a7685dc6d0462f36b440ceb3c9 |
results_ocr_sonnet5_2026-09-20/ (74 files, not tracked in git; manifest hash) |
the Sonnet 5 open-code-review arm’s raw per-PR runs: per-PR elapsed time, including the 6 over-window PRs at 8 to 16 minutes each. Manifest hash: each of the 74 files was hashed with sha256sum, the 74 lines sorted by path, and that combined text hashed again with sha256sum |
7dd843d4c8d25c5afb45ff722ad1321fff508d347363df4f5a4d49428ee60057 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-E__pr-114__run-1.json |
repo E #114 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | 065cb3b90ff18a7567db15df896e281839a63972eae1dafa857cf36eeed2dbb2 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-E__pr-117__run-1.json |
repo E #117 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | 013c7655374b4a6cbfb8057d908174c4505cbe4f7836844beb3eb6c45b002b76 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-E__pr-122__run-1.json |
repo E #122 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | 24cc48687ee3e3836141c5a094487addc83de17d8dfed7a7d92aa5db5851777d |
results_coderabbit_v3_2026-09-20/coderabbit/repo-E__pr-49__run-1.json |
repo E #49 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | e7d993ea63fe49926f982161cd49cb499e9b25bab710a05e505c372adec405f4 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-E__pr-73__run-1.json |
repo E #73 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | c1033d2219054456d230d98b7fc99d4d4759d06dc234742871afccb3935ba8e3 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-E__pr-84__run-1.json |
repo E #84 row of the CodeRabbit per-PR table (comment count, inline-thread count), flagged against the golden set | 90c93f5e56ab7a060499fca010cebe42c6faef1c0864bd27729c3d79c4138e03 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-B__pr-389__run-1.json |
repo B #389 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | ff75283d7bf2b3d399ef0fa3090f9a5a74f141d77cd0a525258d2bf212744c07 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-B__pr-395__run-1.json |
repo B #395 row of the CodeRabbit per-PR table (comment count, inline-thread count), flagged against the golden set | e7abd0516317d10b5a1f9b5e1b9f236152fbfffce98fa490ab07110e0771e12f |
results_coderabbit_v3_2026-09-20/coderabbit/repo-B__pr-454__run-1.json |
repo B #454 row of the CodeRabbit per-PR table (comment count, inline-thread count), flagged against the golden set | ec5ae868c36f5db0f6a8d5b90f7a0f0352239033e2e4f1e2fa59c3abd95198ff |
results_coderabbit_v3_2026-09-20/coderabbit/repo-F__pr-11__run-1.json |
repo F #11 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | f3ff0672fe5f647afdb1c5b8d3d4777c8fc36fb711385e062a36349e903db265 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-F__pr-88__run-1.json |
repo F #88 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | c307d80db7630323ba6021bc2a6bd020d630d23c546d2686c7ccc2c62df0af5d |
results_coderabbit_v3_2026-09-20/coderabbit/repo-C__pr-164__run-1.json |
repo C #164 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | eb95b5a3d931ac2b3cd1c837df25cee9faf6fb0a18399fe557db943a013fdc16 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-C__pr-20__run-1.json |
repo C #20 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | fdb25cbca91a066250d845f0b3d0cf9671b84d8e0193792dc10c0b5eb3498069 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-C__pr-271__run-1.json |
repo C #271 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | 4814874ce084fe54343fe5d7c1c4bfeebf90181b43331589e8d382cc883cedee |
results_coderabbit_v3_2026-09-20/coderabbit/repo-C__pr-491__run-1.json |
repo C #491 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | 86e7748442fb73c3b74fd7b9308024237a4d9c15934eabd34087fa50742b2cc9 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-C__pr-5__run-1.json |
repo C #5 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | 9a083a90d351c48a682732b396cd93e0a0c612d0c6aecfcee73aa99bf607e48c |
results_coderabbit_v3_2026-09-20/coderabbit/repo-C__pr-68__run-1.json |
repo C #68 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | 2bc6c4a576b4f40ee45b8ef710f9829c36603b02bf3fdee31545a6fe36dda431 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-C__pr-69__run-1.json |
repo C #69 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | 434ad40c6b391fe895a0e94fb56f45b084fe418b4a54b08f09ca840b14459ef9 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-D__pr-51__run-1.json |
repo D #51 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | af7aa9848bddd9aa7f088e252d247a2903727f93e0b0a7ec6c8ee57c92202657 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-D__pr-7__run-1.json |
repo D #7 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | a545692e9a22c119fefe1ab15a908cc32f0614b29e4eef192acd7abe7739f1e8 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-A__pr-139__run-1.json |
repo A #139 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | c4e0bb581378ec54028ac2b18ceab0db1cecd961bd39980f81a61ff3d28f49d6 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-A__pr-149__run-1.json |
repo A #149 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | 0d24c335b6a6e1e17b73eb0cc816028f36ee4d4a42497fbc8ed546c17011988a |
results_coderabbit_v3_2026-09-20/coderabbit/repo-A__pr-236__run-1.json |
repo A #236 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | b2b3f4a82f7d4e87fd5d5b3a08707b0a6d10489c14d4ce0c203f646c3630441f |
results_coderabbit_v3_2026-09-20/coderabbit/repo-A__pr-39__run-1.json |
repo A #39 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | 61f8c3804effa1dae5ddcbb8f42c87c243f0bd065c6970cde1e9f6d5ce5b2da8 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-A__pr-46__run-1.json |
repo A #46 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | 8ba9410534f1580eb171549920b9357d3d2ff9104da1ae5844736b19ece4fcc2 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-A__pr-51__run-1.json |
repo A #51 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | fcdf6470719db5290f321a7dde4881122e85bd1188f685b0f11863cdd3fff0a7 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-A__pr-5__run-1.json |
repo A #5 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | 354f71dd59e7b0f2080f76efc0080918845bc0a489f96b51e2a29d5f0c675aa2 |
results_coderabbit_v3_2026-09-20/coderabbit/repo-A__pr-75__run-1.json |
repo A #75 row of the CodeRabbit per-PR table (comment count, inline-thread count), not flagged against the golden set | dec0f3589fd2b737034d37b0436455299cdb7d256d7d20e87064a8991f2c7c1d |
scored_v3_seeded_deep_2026-09-15.json |
seeded-basis scoring: strict 10 of 11 and audited 11 of 11 union recall, per-seed counts, the per-PR seeded-location table | deeef24273da878ded7d36ed8f55a11ea6358d88ba0e93d4e216284b7c869ea8 |
ground_truth_v3/ (90 tracked files, manifest hash) |
Act 1 corpus: 5,215 mined pairs, 137 candidates on 110 PRs, 43 curator keeps, 9 audit overrides, and the finished 47-item / 37-PR / 8-security-39-correctness / 4-critical corpus with its pinned diffs. Manifest hash: each of the 90 tracked files was hashed with sha256sum, the 90 lines sorted by path, and that combined text hashed again with sha256sum |
998b2a43cab38dc796cdf4274550c9e9e299c311a1d76e9d71b5f0a456439dca |
ground_truth_v3_seeded/README.md |
seeded-set corpus description: pipeline, yield (12 planted, 11 in the finder’s window) | 9157da44f1f43e0e34fce80d26fd92c93bda145eb5398caa3cdb1c19c990b76a |
ground_truth_v3_seeded/diffs/repo-E__171.diff |
repo E #171 seeded diff (raw-error-to-user, security / minor) pinned for the seeded item-by-item table | c743b1b4e95ead599e6a4a0189cb4cfb64d56fbd5e173c03d5edc8bc937cbbb1 |
ground_truth_v3_seeded/diffs/repo-E__173.diff |
repo E #173 seeded diff (validator-skips-sibling-field, correctness / major) pinned for the seeded item-by-item table | 5aca5d55ccfdfc44b8bf4ed350f0223fd913bf876a75cab33fc0bf8412e520ec |
ground_truth_v3_seeded/diffs/repo-E__176.diff |
repo E #176 seeded diff (auth-check-missing, security / major) pinned for the seeded item-by-item table | 746894273313ea9bbe7da0dad7c9efb1f1e254de80fd67f6934d63ef8a1f4d03 |
ground_truth_v3_seeded/diffs/repo-B__1057.diff |
repo B #1057 seeded diff (tenant-scope-omitted, security / critical) pinned for the seeded item-by-item table | 8e07f2f1b98901d28cf56b345db35d949fc2f9d51e997a05d1eb2c3c64d7bafa |
ground_truth_v3_seeded/diffs/repo-B__1063.diff |
repo B #1063 seeded diff (tenant-scope-omitted, over-context, excluded from the item table) pinned for the seeded item-by-item table | 5f8bce8dddc1b0149b2e73eeef73c66970ac3062c06ff0c1c07cfa87b3c22a40 |
ground_truth_v3_seeded/diffs/repo-F__130.diff |
repo F #130 seeded diff (falsy-zero-coerced, correctness / major) pinned for the seeded item-by-item table | 5c00c534e514ee7975305873ff0dcc68b062264ccb71a5a8af47ad2734ffdb11 |
ground_truth_v3_seeded/diffs/repo-F__135.diff |
repo F #135 seeded diff (falsy-zero-coerced, correctness / major) pinned for the seeded item-by-item table | d2759bfee173cf815c0b89c65777ec2bbd2110cf78a7ca5b679646e6461ec675 |
ground_truth_v3_seeded/diffs/repo-C__576.diff |
repo C #576 seeded diff (validator-skips-sibling-field, correctness / major) pinned for the seeded item-by-item table | c51286831632685f56accd221d2a1318ae2339c1b84dffac7d1f2fd4b46a1d68 |
ground_truth_v3_seeded/diffs/repo-C__578.diff |
repo C #578 seeded diff (except-too-narrow, correctness / major) pinned for the seeded item-by-item table | a926790b53af1421bab7896cdfd897a1394be1a8d4bf3153c917c28c576d694d |
ground_truth_v3_seeded/diffs/repo-C__585.diff |
repo C #585 seeded diff (validator-skips-sibling-field, correctness / major) pinned for the seeded item-by-item table | 8824bf13908a1bfc50111c6ae8f488c4ebdee51a0fbc9b07dbe308e5a26211ea |
ground_truth_v3_seeded/diffs/repo-C__586.diff |
repo C #586 seeded diff (dedupe-key-too-coarse, correctness / major) pinned for the seeded item-by-item table | 0f7d3f8b8ab024f4bfb946fb4690c1e61a677e90ea87feb18b135fe07607f557 |
ground_truth_v3_seeded/diffs/repo-A__414.diff |
repo A #414 seeded diff (unit-scale-mismatch, correctness / major) pinned for the seeded item-by-item table | a74d0fd6c27042d6de77572d32495f6ed7fd9748b5f18ca641207ff2c9135357 |
ground_truth_v3_seeded/donor_classes.json |
the 15 bug classes distilled from the incidents before planting (the seeded-basis section) | 1ed2923c3afa45a4fbdea73492a14ea3992b5e72d986ea345f41283409817eae |
ground_truth_v3_seeded/manifest.json |
seeded-set pin metadata: base/pin commits and diff sizes for all 12 seeded PRs | 7a026929382c22fc9200c76212aa8c8af1ddf7e395a17a93308a7619ffc3e6e6 |
ground_truth_v3_seeded/repo-E__171.json |
repo E #171 seeded incident record (raw-error-to-user, security / minor): planted location, class and category | 5d8652492cb70d06a8081977f8d5e3c5b0ac81f130718acf20ce16ed881e7277 |
ground_truth_v3_seeded/repo-E__173.json |
repo E #173 seeded incident record (validator-skips-sibling-field, correctness / major): planted location, class and category | a57122f5ae7067b9f263bf9925e205d6936c87b98e65d1f4c487976417129a1a |
ground_truth_v3_seeded/repo-E__176.json |
repo E #176 seeded incident record (auth-check-missing, security / major): planted location, class and category | 47431af7899faccf334cf9b8adf3b2c7d24b814a003a5f81c02874cc6e5e2ff5 |
ground_truth_v3_seeded/repo-B__1057.json |
repo B #1057 seeded incident record (tenant-scope-omitted, security / critical): planted location, class and category | 1a6e10a95dcc551598bbf5d16bc991890352d3b4006e3e72b93ae42dac1bff0b |
ground_truth_v3_seeded/repo-B__1063.json |
repo B #1063 seeded incident record (tenant-scope-omitted, over-context, excluded from the item table): planted location, class and category | 43f0619689aa3aa3adb323d2a3cbc2898a844765b5af6ebd276faeaaef32ca25 |
ground_truth_v3_seeded/repo-F__130.json |
repo F #130 seeded incident record (falsy-zero-coerced, correctness / major): planted location, class and category | bbb26096c4c7c962d33902bc6c460d438798e900bd8659a2f4cd54e616f99383 |
ground_truth_v3_seeded/repo-F__135.json |
repo F #135 seeded incident record (falsy-zero-coerced, correctness / major): planted location, class and category | 755c466429505fd54a9dbd980e0c5c201f548fe8b266672c9015027638db0a2f |
ground_truth_v3_seeded/repo-C__576.json |
repo C #576 seeded incident record (validator-skips-sibling-field, correctness / major): planted location, class and category | 68564fcb62fdc0a970efb7743f839266927213cdb22e2a1a5fa6875eba1f25dc |
ground_truth_v3_seeded/repo-C__578.json |
repo C #578 seeded incident record (except-too-narrow, correctness / major): planted location, class and category | fe956632b7b7e0ff4a65c75f7ae60076dea8fb3a98e36843604c476fb0193a67 |
ground_truth_v3_seeded/repo-C__585.json |
repo C #585 seeded incident record (validator-skips-sibling-field, correctness / major): planted location, class and category | d402e5b1e72c2442aa5be9b8e912e05a3f87fe760304bcaf9ede7acf9ee3cbcc |
ground_truth_v3_seeded/repo-C__586.json |
repo C #586 seeded incident record (dedupe-key-too-coarse, correctness / major): planted location, class and category | a396ebd8bba35bed640e1af296452d140eb981c8418c090f5650b315047fb26e |
ground_truth_v3_seeded/repo-A__414.json |
repo A #414 seeded incident record (unit-scale-mismatch, correctness / major): planted location, class and category | 04f54278f1fdabf87185849df56750ae3188dda7d8f1b0219d4f2b2694ef67df |
ground_truth_v3_seeded/sample.json |
a worked example of a seeded record’s shape | 6cb66bce6717e0ba3b69abd110e7f843ae70a9e2bbf3d2f649365b6cdd242b94 |
ground_truth_v3_seeded/seeded_audit.json |
the hand audit distinguishing strict vs. audited credit in the seeded item-by-item table | 8fd03c75a8000dbe81a1c2b29c3c876b39a7f74adf6335efd5065f0c68513214 |
adjudicated_v3_ceiling3600_2026-09-19.json |
the 3,600-second ceiling run on the four PRs that timed out twice: 2 of 5 items caught vs 1 of 5 at 1,800 s, precision 0.008 vs 0.019 | 06ec682f507d03732b50993d88c5297cba09fcd1985de49e2edabcae6a4bbda4 |
judge_matches_v3_ceiling3600_2026-09-19.json |
the judge decisions behind that run | e350a4b7fda02ef2f1d62fa074507ce1db57c94b9c75d35a719ee56b345df6ef |
Some figures rest on a record written when each run finished, not on a pinned raw file. The thread counts (329 of 564 acted on, and each per-PR ratio) were read from GitHub’s review-thread state and recorded in the tier-2 findings spec; they can be re-derived from the pull requests themselves. The token counts, tool calls, and dollar costs of the triage passes and the open-code-review run are the run history in the 2026-09-20 tier-2 findings spec; the triage file’s own cost field was never populated. The second triage pass kept no per-finding file, because the third pass overwrote it. The seeded basis and the ceiling run are pinned above; the ground-truth corpus is pinned by its manifest hash.
What is withheld
The six repositories are private and are not named; each is a letter, stable across the series. The pinned diffs, the raw finder and triage outputs, CodeRabbit’s comment texts, and the judge rationales quote application source and stay private. File paths, line numbers, verdicts, and paraphrased defect descriptions are on this page.
Written under Spark Bench style v0.5