Qwen3.8 Beat My Driver on Every Test. Nemotron Got the Job.

On 2026-08-19 I retired Qwen3-Next-80B-A3B (NVFP4, vLLM 0.21.0) as the driver of the Hermes agent fleet on the DGX Spark on my desk, because Qwen3.8-27B (NVFP4, vLLM 0.27.1) beat it on every quality check I ran: coding 18 of 18 packets, vision 3 of 3, tool calling 6 of 6, long-context retrieval 12 of 12. Qwen3.8 did not get the seat. It decodes at 11.3 tok/s, 3.4× to 3.8× slower than the model it beat, because 27B dense parameters read more memory per token than 80B with 3B active. A driver is paid per turn, dozens of short decodes and tool calls per task, so the seat went to Nemotron-3.5-Lightning-30B-A3B (NVFP4): 78.8 tok/s, 6 of 6 on the same tool probe, 20 of 20 on multi-turn chains, 3B active parameters. Pick the driver on decode speed and tool chains, measured head-to-head. Quality wins go to the escalation seat, and that is where Qwen3.8 went.

The stack

One NVIDIA DGX Spark, GB10, 128 GB unified memory. Qwen3.8-27B NVFP4 and Nemotron-3.5-Lightning-30B-A3B NVFP4, both under vLLM 0.27.1 as systemd units since 2026-08-19; the incumbent ran under vLLM 0.21.0 until that evening. Claude Opus over the API judged the coding packets; every other scorer is code. CodeRabbit and a paid frontier model keep the jobs the routing table says local models cannot carry.

Act 1: The better model retired the incumbent

The question on 2026-08-18 was whether a new checkpoint replaces the driver. Two days of evaluation, on the box, at $0 marginal except the judge.

check	instrument	Qwen3.8-27B	comparison
coding	18 Python packets, four rungs, frontier judge, retry budget	18 of 18 recovered, 16 first attempt, rung 3 cleared	incumbent’s June verdict: cliff at rung 3
vision	screenshot transcription, exact match	3 of 3	Qwen3-VL-4B sidecar: 3 errors on one image, then a justification for them
tool calling	6-case probe scored on the parsed `tool_calls` array	6 of 6	no baseline arm
long-context retrieval	planted fact in repo source at 10, 50, 90% depth	12 of 12 at 64K and 128,712 tokens	no depth effect

One honest scope line: of the four, only decode was a matched head-to-head against the incumbent on the same prompts. Coding compared a fresh run to a remembered verdict, vision compared against a sidecar, tool calling ran no baseline. It did not matter, because the incumbent had lost on the one thing a driver cannot lose on, and Qwen3.8 was about to lose on it too.

Act 2: It cannot drive

Four prompts, 512 output tokens each, thinking off on both, 2026-08-19:

prompt	Qwen3-Next-80B-A3B	Qwen3.8-27B	penalty
4K	36.9 tok/s	10.3 tok/s	3.6×
16K	28.6	8.5	3.4×
32K	27.4	7.2	3.8×
60K	20.6	5.4	3.8×

The cause is architectural. Decode on this box is bound by memory bandwidth, so per-token cost tracks the bytes of active parameters. Qwen3.8 has dense feed-forward layers; its config has no expert count at all. Qwen3-Next activates about 3B of its 80B per token. “27B is smaller than 80B” points the wrong way, and the number that decides it is on no model card.

The lever that should have fixed it half worked. Qwen3.8 ships a multi-token-prediction head, and speculative decoding at k=3 was lossless on the full 18-packet suite, 18 of 18 in 16m53s instead of 35m59s, 2.13× faster on code. It never shipped, because on this checkpoint speculation requires disabling the vision tower, and the deployed unit has to answer the vision alias that the retired sidecar left behind. Vision won. The deployed rate is 11.3 tok/s, re-measured against the live unit at n=3 on 2026-08-20.

Act 3: What a driver is paid for

An agent loop is dozens of short decodes with a person behind each one: a tool call, a tool result, a next call, a summary. At 10.3 tok/s a 200-token turn is about 19 seconds. At 36.9 it is about 5. Multiply by the turns in a real session and the gap between usable and “I’ll do it myself” is the decode rate, because most turns are easy and quality on any one of them barely moves the outcome.

A hard coding packet, a screenshot read, a code review, or a 258K-token retrieval is one long decode with the person waiting once. Rung 3 packets averaged 211 seconds on Qwen3.8 and it was the only local model that finished them correctly. The correct four-minute answer beat the wrong one-minute answer. Speed is paid per turn. Quality is paid per task. The two models sit on opposite sides of that ledger, so the driver decision was never Qwen3.8 against the incumbent. It was Qwen3.8 against whichever fast model could survive a multi-turn tool chain.

Act 4: Nemotron

Nemotron-3.5-Lightning-30B-A3B is a mixture-of-experts checkpoint that activates 3B of its 30B parameters per token, served here as NVFP4. It came out of a one-day audition for the fast seat that also tried gpt-oss-20b and gemma-4-26B, every verdict computed in code against stub tools; the routing article has that audition and the flag policy it produced. Against Qwen3.8 on the driver’s job:

	Nemotron-3.5-Lightning-30B-A3B	Qwen3.8-27B
decode, same prompts, 2026-08-20	78.8 tok/s	11.3 tok/s
single-turn tool probe	6 of 6	6 of 6
multi-turn tool chains, 5 scenarios	20 of 20 across four runs	not entered
active parameters	3B of 30B	27B dense

Qwen3.8 matched it on tool calling and lost the seat on speed alone, by 7×. I never ran Qwen3.8 through the chain probe, because a driver at 11 tok/s was already out and the probe would have measured a seat it could not hold. That is a gap in the record, and it is the honest shape of the decision: speed disqualified before quality was asked.

The seat gave things up, and they are measured too. On long-context retrieval with decoys present, Nemotron returned a decoy instead of the planted fact at 90% depth at 256K tokens, and scored 8 of 12 across a 160K to 225K grid with no clean threshold; Qwen3.8 was 12 of 12 at 64K and 128K and clean at 258K with two decoys. As a single-prompt code reviewer on the audited golden set, Nemotron found none of the findable known bugs where Qwen3.8’s deep pass found 0.571. Nemotron has no vision path. And it was never run on the coding packets, so its coding is a hole in the record rather than a loss. The driver cannot be trusted with a long document, a screenshot, or a review. Every one of those turns is handed to Qwen3.8, and the price of the trade is one hand-off and one slow decode per hard task, instead of a slow decode on every turn.

The decision

The pair went live under systemd the evening of 2026-08-19. Nemotron on port 8012 drives Hermes: every agent loop, every tool call, ten short-output auxiliary roles. Qwen3.8 on port 8011 is the escalation model: hard coding, vision, long-document retrieval, and the deep code review pass, the four places the driver measured worse or was never measured. Qwen3-Next-80B-A3B and the Qwen3-VL-4B sidecar were masked, and about 90 GiB of memory came back. The number that decided it is 78.8 against 11.3.

Two consequences carry forward. The 2.13× speculation gain is real and undeployed. On this checkpoint vLLM loads the speculation head only with --language-model-only, which drops the vision tower, and the one Qwen3.8 process also answers the vision alias that the retired sidecar left behind. A second, language-only instance would fix it, and its weights are 22 GB on a box with 22 GB free after the two KV pools, so it fits only by cutting the pool that long-context review depends on. Two ways around it are queued: a vLLM 0.29 trial, in case the newer build lifts the restriction, and an audition for a small vision model, 6 to 10 GB, that passes the probe the 4B failed. If either lands, Qwen3.8 goes language-only with speculation on, and the escalation model’s longest jobs, a four-minute coding packet and a deep review pass that thinks for 10K to 15K tokens, run close to twice as fast. And the driver is a model whose Mamba-2 path has crashed once under sustained batches, so it runs with restart-always and a concurrency cap, which the concurrency article measures.

The playbook

Route by which ledger the workload pays. Agent loops pay decode per turn; hard coding, vision, review, and retrieval pay quality per task.
Pick the driver head-to-head on decode and multi-turn tool chains. Single-turn probes and quality scores do not see the seat.
Active parameters, not total, predict decode on a bandwidth-bound box. Read the config for an expert count before the size in the name.
A lever measured in isolation is not a lever deployed. Check what the unit’s flag set allows before quoting the speedup.
Count matched comparisons before counting wins. Write the scope line into the decision record.

This is Spark Bench: what one DGX Spark on a desk can and cannot be trusted to do, measured honestly. Next: the escalation model in its reviewer’s chair, against 47 bugs that actually shipped in my repos. How many it found, and what that cost, is the next article. Follow for the count.

Appendix: the measurements behind the decision

Everything here was measured on one DGX Spark (GB10, 128 GB unified memory) on 2026-08-18 through 20. Coding verdicts came from Claude Opus over the API, 36 calls, one per packet per arm; the harness never logged their token usage, so that dollar figure does not exist. The frontier executor that takes over a packet when the local model exhausts its retries fired zero times. Every other scorer is code.

A. Coding: the local-executor eval, Qwen3.8-27B, no speculation

rung	packets	first-pass	recovered within retries	mean wall s
r0	3	0.667	1.0	46.2
r1	5	1.0	1.0	114.6
r2	5	1.0	1.0	78.6
r3	5	0.8	1.0	210.8

The incumbent’s June verdict was “Python GO through R2”, cliff at R3. The June run files survive, so a stored diff exists; the executor-cliff article re-measures both current models on the same packets.

B. Decode and speculation

Decode, thinking off, 512 output tokens, input tokens incumbent / candidate:

prompt	input tokens	Qwen3-Next-80B-A3B	Qwen3.8-27B	penalty
4K	4,356 / 4,687	13.9 s · 36.9 tok/s	49.8 s · 10.3 tok/s	3.6×
16K	20,909 / 22,547	17.9 s · 28.6 tok/s	60.0 s · 8.5 tok/s	3.4×
32K	36,941 / 39,866	18.7 s · 27.4 tok/s	70.9 s · 7.2 tok/s	3.8×
60K	64,307 / 69,744	24.9 s · 20.6 tok/s	95.4 s · 5.4 tok/s	3.8×

Qwen3.8’s larger vocabulary emits 7.6 to 8.5% more tokens for identical text, so it pays the slower rate on a longer bill.

Speculative decoding on Qwen3.8, same prompts, wall seconds:

prompt	no MTP	k=1	k=2	k=3	incumbent, no MTP
4K	49.8	30.3	29.0	27.4	13.9
16K	60.0	42.1	35.9	33.1	17.9
32K	70.9	54.1	47.4	46.4	18.7
60K	95.4	79.7	72.2	73.8	24.9

Conditional acceptance 77.3, 73.8, 71.3% at draft positions 0, 1, 2, averaging 2.75 tokens per forward pass at k=3; the KV and prefill tax of each draft token is why k=3 goes backwards at 60K. Accuracy at k=3: 18 of 18, identical to no-MTP, 16m53s against 35m59s. Long-document reads paid about 6% at 64K and 128K; the needle grid stayed 12 of 12. The incumbent had run without its own MTP head for months because a vLLM 0.21.0 weight-loader bug blocked it; 0.27.1 carries the fix.

Prefix caching: 95.8% hit rate at 8K of shared prefix, 1.00× wall-clock speedup at that size, about 7% of the KV pool. Prefill runs near 25,000 tokens per second here, so caching saves nothing until prefill dominates past 32K, and it does not touch decode.

C. Vision, tool calling, retrieval

Vision, Qwen3.8 against the Qwen3-VL-4B sidecar, thinking off: 3 of 3 exact for Qwen3.8 including a branch name and an error string with its comma intact, latency 12 to 18 s. The 4B made three reproducible transcription errors on one screenshot (branch name altered, a test_ prefix dropped, a comma dropped), then asserted the screenshot contained a typo. Latency 3.5 to 6.7 s.

Tool calling, six cases with a no-tool control, scored on the parsed tool_calls array: 6 of 6 with thinking on, 5 of 6 with thinking off, zero parser misses. The deployed unit runs no reasoning parser, so thinking on leaks chain-of-thought into content; the standing policy sends enable_thinking: false to it, which produced one false failure during the vision evaluation before it was understood.

Long-context retrieval: a fact no model could infer, planted in real repository source at 10, 50, and 90% depth, exact match in code: 12 of 12 at 64K and at 128,712 tokens under both the MTP and no-MTP arms. After the serving cap was restored from 131,072 to the checkpoint’s 262,144 (the lower number had crept in during the hand-launch to systemd transition), a needle at 258,431 tokens with two decoys present was retrieved at 330 s per case, at a cost of 10% of the KV pool, because dense attention keeps per-token state.

D. The audition Nemotron came from

Single-turn tool probe and the five-scenario multi-turn chain probe, all verdicts in code against stub tools with non-round values, 2026-08-19:

candidate	single-turn	multi-turn chains	decode	verdict
Nemotron-3.5-Lightning-30B-A3B	6 of 6	19 of 20, re-scored 20 of 20 across four runs after a scorer defect was fixed 2026-08-20	76.7 tok/s hand-launched, 78.8 under systemd	winner
gpt-oss-20b	6 of 6	13 of 15	45.9 tok/s	leaks format markup into tool names
gemma-4-26B-A4B	not reached	not reached		cannot start on this vLLM

The one Nemotron miss was the scorer’s: a phrase list that accepted one correct refusal and rejected its paraphrase. Full audition, flag policy, and routing table: the routing article.

E. Incidents that changed nothing

The 2026-08-18 smoke test, a hand-run vllm serve from a fresh venv, re-triggered the JIT-compile out-of-memory failure the systemd units guard against with thread caps; the box was unreachable for about 19 minutes. Re-run with the caps, vLLM 0.27.1 served NVFP4 on GB10 with stock kernels: weights loaded in 128 s, ready in about four minutes, peak 80 GiB. The forum consensus that GB10 needs a specialized container did not hold for this checkpoint.

F. Deployment record, 2026-08-19 evening

vllm-nemotron.service on port 8012, gpu-memory-utilization 0.25, KV pool 2,777,998 tokens; vllm-q38.service on port 8011, utilization 0.45, KV pool 953,661 tokens, no MTP, serving the vision alias. Qwen3-Next-80B-A3B and Qwen3-VL-4B masked, not stopped. Memory after: about 100 GiB used of 121. Restart-always on the Nemotron unit since the 2026-08-23 Mamba-2 crash under sustained 12-way batches (vllm-project/vllm #37431, still open as of 2026-09-17).

What is not published, and why

The coding packets’ source and the judge rationales quote application code and stay private. Packet ids, rungs, pass counts, and wall times are above. Nothing else is withheld.