Spark Bench · Article 3
Two Local LLMs Replaced One Big One as My Coding Agent
· updated
Correction, 2026-09-28. Two claims below were incomplete. Act 1 says the 2026-08-20 scorer fix settled Nemotron’s one miss. It settled that phrasing, not the defect: on 2026-09-22, re-run on live Nemotron with the corrected scorer, the refusal list failed 2 of 4 correct answers on the same scenario (“I wasn’t able to find any revenue recorded…”, “there’s no recognized revenue on file…”). The model was right 4 of 4. The error-recovery verdict now rests only on what code can prove (the tool was called, it errored, no fabricated figure appears, an answer was given), and the wording goes to a human as a review flag that never fails a run.
gpt-oss-20b’s chain score of 13/15 cannot be re-audited. It was scored before the four scorer fixes of 2026-08-20, one of which could pass a wrong answer and three of which could fail a right one; only Nemotron was re-run afterward, and the per-attempt outputs were not kept. So the figure is unverified in both directions. Its verdict does not rest on it: the tool-name leak in Appendix D disqualifies it on its own. No ranking here changes. The harness we now run this probe on keeps every run’s full raw output.
On 2026-08-20 I retired Qwen3-Next-80B-A3B as the driver for every agent on the DGX
Spark on my desk and replaced it with two smaller models resident at once:
Nemotron-3.5-Lightning-30B-A3B (NVFP4) for tool calls and agent loops, Qwen3.8-27B
(NVFP4) for code and vision. The one-day audition that picked them cost $0 in API
calls, every verdict computed in code against stub tools. Nemotron won the loop seat on
a multi-turn chain probe the single-turn probe could not see, two candidates failed
before quality was measured, and the enable_thinking: false rule my harness sent to
every model was costing the winner a third of its tool calls. Which model gets which
job is a measurement, and the model card’s biggest number predicted almost none of
it.
The stack
One NVIDIA DGX Spark (GB10, 128 GB unified memory), both models under vLLM 0.27.1 behind OpenAI-compatible endpoints. Hand-launched for the audition on 2026-08-19, systemd units from 2026-08-20. No cluster, no cloud GPUs, no fine-tuning, no LLM judge anywhere in the audition. CodeRabbit and a paid frontier model keep the jobs the routing table says local models cannot carry. The serving article covers the memory math and uptime of the pair; the Qwen3.8 article covers why the model that won on quality is not the driver. This one is the routing.
Act 1: Single-turn probes cannot see the defect that matters
The tool-calling probe we already had scored one call: give the model tools, ask a question, check that it picks the right function with valid arguments. Nemotron 6/6, gpt-oss-20b 6/6, and on that instrument the audition would have been decided by decode speed.
An agent driver does something harder. It has to take a tool result back, chain into the next call, and answer from what the tool returned instead of from its own imagination. So we added a multi-turn probe: five scenarios against stub tools, every verdict computed in code, separating a wrong chain from an ungrounded answer from a fabricated number from over-execution from a failure to recover after a tool error. The stub values are non-round (412,837 / 388,519 / 147,206) so a fabricated answer cannot coincidentally match.
That distinction paid for itself on the first run. gpt-oss-20b, perfect on single turns, scored 13/15 on chains. Nemotron-3.5-Lightning-30B-A3B (NVFP4, vLLM 0.27.1, hand-launched, 2026-08-19) scored 19/20 plus a focused 4/4 re-check, at 76.7 tok/s decode against gpt-oss’s 45.9.
| candidate (2026-08-19) | single-turn | multi-turn chains | decode | verdict |
|---|---|---|---|---|
| Nemotron-3.5-Lightning-30B-A3B | 6/6 | 19/20 (+4/4 re-check) | 76.7 tok/s | winner |
| gpt-oss-20b | 6/6 | 13/15 | 45.9 tok/s | structural defect |
| gemma-4-26B-A4B | n/a | n/a | n/a | cannot start |
The one Nemotron miss was ours. The failing scenario was error recovery: the tool returns an error and the model must say it has no data rather than invent some. Nemotron alternated between “there’s no revenue recorded” and “I don’t have any revenue data on file.” Both are correct refusals. Our hand-written phrase list accepted only the first. A PR review caught that scorer defect the next day (2026-08-20), along with three others, and we re-ran against the live units: 20/20 across four consecutive runs, decode re-measured at 79.2 tok/s with generation pinned to a fixed 512 tokens. The model never had the defect the audition charged it with.
Act 2: The two that failed before quality was a question
gemma-4-26B could not start. vLLM 0.27.1 with transformers 5.15 dies at engine init
because this checkpoint’s head_dim genuinely varies by layer: 256 for its 25
sliding-attention layers, 512 for the 5 full-attention layers. The config offers a flag
to force the ambiguity away. We did not use it, because vLLM’s homogeneous assumption
would size the KV cache wrong for exactly the layers that carry long-range context,
which is the work I would be buying the model for. A model you cannot serve correctly
has no score. It has a version-bump ticket.
gpt-oss-20b leaked format markup into tool names. On the second turn of a chain it
returned the function name send_slack_dm<|channel|>commentary, a harmony-format
channel marker fused into the identifier. No dispatcher can route that. It reproduced
at 8 of 24 runs without the thinking kwarg and 6 of 24 with it; a separate n=12 pair
read 6/12 versus 1/12, so the honest statement is that the leak is intrinsic at roughly
one call in three and the kwarg has no reliable effect. Two earlier single runs had
pointed in opposite directions. n=1 on a stochastic defect is worse than no
measurement, because it feels like one.
gpt-oss also fetched a tokenizer vocab at first request rather than at startup, so the engine booted clean and then returned 500 on every completion; that one is a solvable trap. The tool-name leak is disqualifying for an agent driver unless the dispatcher sanitizes names, which is a bug I would be adopting on purpose.
Neither failure is about quality. Both are about whether the model works as a component in the stack you actually run, and both are invisible from a leaderboard.
Act 3: The house rule that cost the winner its tool calls
Both probes had been sending enable_thinking: false to every model, unconditionally.
That is a Qwen convention, inherited from the 80B era, when the driver reasoned by
default with no reasoning parser attached and thinking-on produced malformed output.
Measured on Nemotron (2026-08-19, 6-case tool probe):
chat_template_kwargs |
tool calls passed |
|---|---|
enable_thinking: false |
4 / 6 |
enable_thinking: true |
6 / 6 |
| omitted | 6 / 6 |
The flag we sent as a safety measure was the thing breaking tool calls. Nemotron
reasons by default and cannot usefully be told not to; with --reasoning-parser nemotron_v3 on the serving side its reasoning lands in its own field and never touches
content. Without the parser the answer arrives as "Need to answer.</think>The cannabis pod's…".
The blanket rule was wrong in the opposite direction for short outputs. Because
Nemotron reasons first, a tight max_tokens is spent entirely on reasoning and the
request returns finish_reason: length with content: null. A silent empty result,
no error. On the live endpoint, “Reply with exactly: OK” at 16 tokens came back null
with the flag omitted and OK with it sent; at 512 tokens it was fine either way.
Title generation, classification, scoring and yes/no calls all have this shape, and my
agent config has ten such roles.
And it was wrong a third way on the other endpoint. Qwen3.8 serves vision on a unit with no reasoning parser, so with thinking on, raw chain-of-thought lands in content. During the 2026-08-19 vision evaluation it returned “The user wants a description of the image and the exact values for each bar. The image is a bar chart titled…” and ran out of budget before answering. With the flag sent: 3/3 exact, in half the time.
So the policy became per role and per output shape, written down with the exact syntax:
| consumer | endpoint | thinking flag | why |
|---|---|---|---|
agent loop (model.default) |
Nemotron | omitted | needs tool calls |
| 10 short-output aux roles | Nemotron | false |
empty-result risk beats tool-call loss |
| vision | Qwen3.8 | false |
no reasoning parser on that unit |
| review finder/verifier | Nemotron | false |
structured output; inherited, not re-measured |
The last row is an admitted debt: the review gate’s flag was never A/B’d against Nemotron. It stays off because off cannot produce an empty result, and the code-review benchmark article is where that gate’s real ceiling gets measured.
Act 4: One box, not one GPU each
The pair shares compute as well as memory, and the measurement that proved it belongs to the serving article: with both engines generating at once, Nemotron fell from 77.5 to 28.2 tokens per second (2026-08-19) while Qwen3.8 barely moved, and the box produced less total work than the fast model alone. An earlier note in our own log had recorded “no co-residency penalty,” measured one model at a time.
For routing, that number is the constraint. The pairing is sound for the workload it was built for, which is alternation: an agent turn on the fast model, then a coding turn on the careful one. Any design that has both models talking in the same second is buying a 64% haircut on the driver, and the routing table below assumes they take turns.
The routing table, and where the frontier still gets paid
With the pair live, every task shape in my agent fleet got a row, and every row cites the measurement that put it there. Condensed from the 2026-08-21 routing document:
| task shape | route | evidence |
|---|---|---|
| agentic tool-driving | Nemotron, local | 20/20 chains, ~7× decode (audition, corrected 08-20) |
| structured extraction, classification, judging over rows | local, per-role model | pacing-judgment set: Qwen3.8 17/20 either sampling arm |
| embeddings, retrieval, rerank | local bge-m3 | settled in an earlier phase |
| vision | Qwen3.8, local | 3/3 exact vs the 4B sidecar’s errors |
| code generation | local up to a measured complexity cliff, frontier beyond | executor eval; the cliff moved in August, the executor-cliff article’s story |
| deep code review, subtle-bug recall | CodeRabbit kept, frontier escalate | the code-review benchmark article |
| open-ended synthesis, hard planning | frontier, paid | local wins never generalized to synthesis |
Two rows retired something faster or cheaper, and both were decided by a probe rather
than by taste. The vision row dropped the Qwen3-VL-4B sidecar: it answered in a third
of Qwen3.8’s time, made three reproducible transcription errors on one screenshot, and
then invented a justification for them (Appendix G). The extraction row kept
presence_penalty: 0.0 against the card’s recommended 1.5, because the two settings
tied on the pacing-judgment set and the harness’s own decision rule calls the one-case
difference on a variant noise (Appendix H).
The rows that route to a paid frontier model are the ones I would most like to flip and cannot. On a live PR the paid reviewer found five real hardenings that neither local model raised. Planning and open-ended synthesis have no golden set yet, and the one big-model experiment there, a 120B Nemotron as planner, was not better than the workhorse of the day (Appendix I).
A model swap re-opens every row it touches. A “local beats Sonnet” headline from May went stale the day its model was retired, so each row is bound to the model actually serving, with the eval to re-run named alongside.
The playbook
- Probe the chain, not the call. If the job is an agent loop, feed a tool result back and score the whole chain in code against stub values nobody could guess.
- Fitness to serve is a gate before quality is a question. A model that cannot start, or returns 500 after a clean boot, has no score.
- Flag policy is per model family, per role, per output shape. Write it as a table with the endpoint and the shape, and date it.
- When a model fails one scenario, read the scorer first.
- Two engines on one box share compute as well as memory. Measure with both generating at once before claiming isolation, and route so the pair alternates.
- A card-recommended setting is an A/B arm, not a default.
- Bind every routing row to the model serving it, and name the re-run. A new checkpoint should cost a day of re-measurement, not a quarter of folklore.
This is Spark Bench: what one DGX Spark on a desk can and cannot be trusted to do, measured honestly. Next: the careful model in this routing table scored 18 of 18 on my coding eval and still did not get the driver seat. Why a perfect score lost is the next article. Follow for it.
Appendix: the numbers behind the routing
Everything in this section is taken from the DGX configuration log and the eval artifacts it points to, stamped with the date each number was measured. The audition and the thinking-flag probes used no LLM judge; every verdict was computed in code. The one experiment below that did use a paid judge (the planner comparison) is marked as such.
A. Serving stack, as audited and as deployed
Both models are NVFP4 checkpoints under vLLM 0.27.1 on one DGX Spark (GB10, 128 GB unified memory, no MIG, no MPS). Two configurations matter: the hand-launched audition engines (2026-08-19) and the systemd units that replaced them (2026-08-19 evening, re-verified 2026-08-20).
| engine | port | role | gpu-memory-utilization | KV pool | concurrency at 131K ctx | parsers |
|---|---|---|---|---|---|---|
| Nemotron-3.5-Lightning-30B-A3B, audition | :8012 | tool driver | 0.30 | 4,372,707 tok (14.36 GiB) | 33.4× | reasoning nemotron_v3 |
| Qwen3.8-27B, audition | :8011 | coding + vision | 0.30 | 221,466 tok (7.05 GiB) | 1.69× | tool-call qwen3_xml, no reasoning parser |
| Nemotron, rebalanced (same day) | :8012 | tool driver | 0.22 | 1,725,781 tok (5.67 GiB) | 13.2× | as above |
| Qwen3.8, rebalanced | :8011 | coding + vision | 0.38 | 613,175 tok (19.49 GiB) | 4.68× | as above |
vllm-nemotron.service (deployed) |
:8012 | driver, model.default + 10 aux roles |
0.25 | 2,777,998 tok (9.12 GiB) | 21.2× | as above |
vllm-q38.service (deployed) |
:8011 | escalation: coding, and the vision alias |
0.45 | 953,661 tok (30.33 GiB) | 7.28× | no MTP (mutually exclusive with the vision alias) |
Why the equal 0.30 split was wrong: Nemotron is a Mamba hybrid whose layers mostly carry constant-size state rather than per-token KV, so the same memory buys about 304K tokens per GiB on Nemotron and about 31K on Qwen3.8. An even split hands the tool driver roughly 20× more concurrency than it can use and starves the model doing the long-context work. Peak memory during the paired bring-up was 108 GiB of 121; the deployed pair sits at 100 GiB used, 20 free.
Retirement accounting (2026-08-19): stopping and disabling the 80B incumbent and the Qwen3-VL-4B vision sidecar took the box from 106 GiB used to 87 GiB, with both audition engines still running. Qwen3.8’s KV pool grew 221,466 → 613,175 → 953,661 tokens across the two retirements.
B. The multi-turn chain probe
Five scenarios against two deterministic stub tools: a revenue lookup that takes a pod name and a month, and a Slack direct-message sender. Stub returns are non-round on purpose (412,837 / 388,519 / 147,206) so an answer that carries one of them proves the model read the tool result. Sampling temperature 0.2; a hard cap on steps per chain; every verdict computed in code.
| scenario | what a correct run must do | what it must not do |
|---|---|---|
| single lookup | call the lookup once, answer with the returned figure | send a DM |
| two-step chain | look up, then send a DM whose body carries the returned figure, in that order | skip either call or reorder them |
| conditional negative | look up, then decide from the value that the DM condition is not met | send the DM |
| error recovery | the tool returns an error; say there is no data | invent a figure (any of the three seeded values, or any new number) |
| no tool needed | answer directly | call any tool |
Verdict classes, scored separately: wrong chain (tools in the wrong order or missing), ungrounded (answer does not carry the tool-returned value), fabricated (a seeded or invented number appears where the tool returned an error), over-execution (a forbidden tool called, or a write repeated), no recovery (the error path not handled). A repeated read is call churn, a repeated write is a second real-world action; only the first is collapsed as a retry.
Results, all candidates:
| candidate | date | single-turn (6 cases) | chains | decode | verdict |
|---|---|---|---|---|---|
| Nemotron-3.5-Lightning-30B-A3B | 2026-08-19 | 6/6 | 19/20, plus 4/4 focused re-check | 76.7 tok/s | winner |
| Nemotron, co-resident after rebalance | 2026-08-19 | 6/6 (Qwen3.8 side) | 5/5 | 78.8 tok/s | holds |
| Nemotron, under systemd | 2026-08-19 | 5/5, 5/5, 4/5 (the miss is the error-recovery scenario) | consistent with 19/20 | ||
| Nemotron, corrected scorer | 2026-08-20 | 20/20 across 4 consecutive runs | 79.2 tok/s (n=3, fixed 512-token generation) | under-reported, not flaky | |
| gpt-oss-20b | 2026-08-19 | 6/6 | 13/15 | 45.9 tok/s | structural defect (see D) |
| gemma-4-26B-A4B | 2026-08-19 | cannot start on vLLM 0.27.1 + transformers 5.15 |
Four decode figures for Nemotron appear in the log. They are four setups, not a contradiction, and the spread is about 3%, inside run-to-run variance:
| figure | setup |
|---|---|
| 76.7 tok/s | audition, hand-launched at util 0.30, co-resident with the audition Qwen3.8 engine |
| 77.5 tok/s | solo, neighbour idle (the baseline of the contention table in F) |
| 78.8 tok/s | under systemd at util 0.25, neighbour idle, EOS-terminated generation |
| 79.2 tok/s | same, generation pinned to a fixed 512 tokens via min_tokens and ignore_eos |
Qwen3.8’s decode on the same box: 11.3 tok/s (n=3, 2026-08-20), 11.4 solo in the contention run. The ~7× ratio holds on every pairing.
C. The four scorer defects (found in review of the audition PR, fixed 2026-08-20)
| defect | effect | fix |
|---|---|---|
| grounding check matched digit fragments | “$412 across 837 campaigns” scored as grounded | match the whole value only |
| number flattening stripped all whitespace | adjacent numbers welded into one digit run, so a correct answer failed the boundary check | strip only commas between digits |
| consecutive retry of the same tool scored as wrong chain | masked the separate call-churn signal | collapse consecutive duplicates first |
| refusal phrase list too narrow | “I don’t have any revenue data on file” rejected; “there’s no revenue recorded” accepted | added the paraphrases |
The fourth defect is the whole story of the audition’s one miss. Fabrication is caught independently by the forbidden-digits check, so widening the phrase list did not weaken the test.
Amended 2026-09-28. The fourth fix did not hold. A whitelist of correct
refusals is never complete, and on 2026-09-22 it failed 2 of 4 correct answers.
The phrase check was demoted to a review flag (commit 4fd7d20); playbook item 4
stands, with the corollary that a list of acceptable answers should never decide
a verdict on its own. The four 2026-09-22 runs, raw (three in the first file, one in the second):
| file | what it holds | sha256 |
|---|---|---|
2026-09-22T22-40-29-00-00_driver-probe_G9yT7sSyoskaNTTLjga5n7.eval |
error recovery × 3: two passes, one false fail (“I wasn’t able to find…”) | 1cdf42ece9e87ac2bd3d459a822f7d961aead7f4bdb52171e44432988321f1e1 |
2026-09-22T22-41-30-00-00_driver-probe_f24gK7LY5JuHM8DV22vNgt.eval |
error recovery × 1: false fail (“there’s no recognized revenue on file…”) | 0fb141b5bc4386efac41ee673361fc3d5443c83bd8f8613fc0d7859fc711b3ca |
D. gpt-oss-20b, the tool-name leak
On the second turn of a chain the model returned a function name with a
harmony-format channel marker fused into it (send_slack_dm<|channel|>commentary).
Reproduction counts:
| arm | leaks / runs |
|---|---|
| without the thinking kwarg, n=24 | 8 / 24 |
| with the thinking kwarg, n=24 | 6 / 24 |
| separate pair, n=12 each | 6 / 12 vs 1 / 12 |
The two pairs disagree on the kwarg’s direction, so the honest statement is a leak rate of roughly one call in three and no reliable effect from the flag. Two earlier single runs had pointed in opposite directions. A second, solvable trap: the harmony renderer fetches its tokenizer vocab at first request rather than at startup, so the engine boots clean, serves the model list, and then returns 500 on every completion until the vocab is available locally.
gemma-4-26B: engine init dies because the checkpoint declares head_dim: 256
for its 25 sliding-attention layers and 512 for its 5 full-attention layers,
which transformers 5.15 reports as an ambiguous per-layer attribute. The
override flag was not used, because vLLM’s homogeneous assumption would size
the KV cache wrong for exactly the full-attention layers.
E. The thinking flag, every measurement
Nemotron, 6-case single-turn tool probe, 2026-08-19:
chat_template_kwargs |
tool calls passed |
|---|---|
enable_thinking: false |
4 / 6 |
enable_thinking: true |
6 / 6 |
| omitted | 6 / 6 |
Nemotron, short outputs on the live endpoint (it reasons first, so a tight budget is spent entirely on reasoning):
| request | flag | result |
|---|---|---|
max_tokens: 16, “Reply with exactly: OK” |
omitted | finish=length, content=null |
| same | false |
OK |
max_tokens: 24, four-word title |
omitted | finish=length, content=null |
| same | false |
a four-word title |
max_tokens: 512, “Reply with exactly: OK” |
omitted | OK |
Qwen3.8, same 6-case tool probe, 2026-08-19: 5/6 with thinking off, 6/6 with thinking on, zero parser misses, all arguments valid JSON. The recovered case was an indirectly phrased request. So the house rule inverted on Qwen3.8 for tool calls too; it stays off there for a different reason (no reasoning parser on that unit, so thinking-on lands in content), and the coding and vision numbers were all taken with it off.
Qwen3.8 vision with thinking on, 2026-08-19: the model narrated its plan (“The user wants a description of the image and the exact values for each bar…”) and exhausted the budget before answering. With the flag: 3/3 in half the time.
The resulting policy table is the one in the body. Its last row (the review finder and verifier on Nemotron, flag off, inherited) is the only row not backed by a measurement.
F. Co-residency: memory is isolated, compute is not
Same prompt on both engines, 2026-08-19:
| engine | solo (neighbour idle) | both generating at once | delta |
|---|---|---|---|
| Nemotron | 77.5 tok/s | 28.2 tok/s | −64% |
| Qwen3.8 | 11.4 tok/s | 11.3 tok/s (min 7.2) | flat at the median |
Aggregate under contention is about 40 tok/s combined against 77.5 for Nemotron alone: simultaneous load yields less total work than the tool driver by itself. Mechanism: the GPU reports no MIG partitioning and no MPS daemon, so the driver time-slices between the two CUDA contexts, and Qwen3.8’s dense decode issues long bandwidth-saturating kernel sequences that Nemotron’s short MoE steps wait behind. An earlier log entry recorded “no co-residency penalty (76.7 → 78.8)”; it was measured one model at a time and describes memory only.
G. The vision probe that retired the 4B sidecar (2026-08-19)
Three images, one run each, both models with thinking off:
| prompt | Qwen3.8 latency | 4B sidecar latency | Qwen3.8 | 4B sidecar |
|---|---|---|---|---|
| bar chart, read every value | 18.1 s | 5.6 s | all four values exact | all four values exact |
| CI-failure screenshot, extract build, branch, test, error | 12.2 s | 6.7 s | exact, including the branch name and the error string with its comma | three errors: branch name mis-transcribed, test path lost its test_ prefix, comma dropped; then asserted the screenshot contained a typo |
| five shapes, position and colour | 17.0 s | 3.5 s | 5/5 | 5/5 |
The sidecar was faster on every image and wrong on the one that looked like real work. The confabulated justification is the disqualifier: a model that explains away its own transcription error will file the wrong ticket with confidence.
H. The card-recommended sampling knob (2026-08-19)
Pacing-judgment set, 20 cases, Qwen3.8, presence_penalty 0.0 (as run) versus
1.5 (the model card’s recommendation), plus a withheld-context variant of each:
| arm | pass rate | action accuracy | guardrail clean | wrong-action failures |
|---|---|---|---|---|
| majority baseline | 0.30 | |||
| as run (0.0) | 0.85 (17/20) | 0.85 | 1.00 | 3 |
| recommended (1.5) | 0.85 (17/20) | 0.85 | 1.00 | 3 |
| as run, withheld context | 0.70 (14/20) | 0.70 | 0.95 | 6 |
| recommended, withheld context | 0.75 (15/20) | 0.75 | 0.95 | 5 |
Per stratum: both full-context arms 5/7 on the enriched cases and 12/13 on the pilot cases, with identical per-case failures. The harness’s decision rule classifies every pairwise delta (0, 2, 3 cases) as noise. Production config unchanged.
I. The one big-model planning experiment (2026-06-09, older stack)
This predates the audition and used a paid judge, so it is not part of the $0 claim. Nemotron-3-Super-120B (reasoning off and on) against the 80B incumbent on 6 repo-grounded decomposition tasks, 3 runs each (n=18 per model), vLLM 0.21.0, deterministic structural floor plus a Sonnet 1–5 rubric:
| metric | 120B, reasoning off | 120B, reasoning on | 80B incumbent |
|---|---|---|---|
| structural pass | 1.00 | 0.944 (invented a file handle) | 1.00 |
| coverage (1–5) | 2.94 | 2.61 | 3.17 |
| ordering (1–5) | 4.39 | 4.00 | 4.11 |
| actionability (1–5) | 3.33 | 3.39 | 3.28 |
| padding steps (lower is better) | 0.33 | 0.11 | 0.83 |
| latency, mean / median | 13.2 s / 12.5 s | 55.8 s / 52.3 s | 5.7 s / 5.5 s |
| completion tokens, mean | 196 | 851 | 217 |
Preliminary by its own write-up (small n, single judge pass), but the direction was enough to revert the 120B cutover: it was not better, and it was 2.3× to 10× slower. That is why the routing table sends hard planning to a frontier model rather than to a bigger local one. An earlier version of this sentence promised a follow-up article re-measuring planning on the current pair; that article was dropped (2026-09-28), and the 2026-09-19 re-run stays in the lab’s config log.
What is not published, and why
The probe scripts, the Hermes profile and toolset configuration, and the per-run transcripts are not on this page. The scripts are source code of the lab’s own tooling; the transcripts are regenerable and were never committed; the profiles carry business routing. Every scenario, stub value, sampling setting, unit parameter, and verdict count they produced is above. Raw per-run records for the planner experiment were likewise regenerable and uncommitted; the aggregate table is the durable record.
Written under Spark Bench style v0.2