Spark Bench · Article 2
Two Local LLMs Serve My Coding Agents From One DGX Spark
I keep a personal agent fleet, and all of it runs on one NVIDIA DGX Spark on a desk at home. No cluster, no cloud GPUs. What I wanted from it was specific: two local models resident at the same time, a fast one to drive the agents’ tool loops and a careful one for hard code and screenshots, both up around the clock, both coming back after a reboot without me.
My Claude Code sessions and I have kept a dated change log of every vLLM flag on that box since 2026-05-20. It is 240 KB long. Most of it is about memory, and memory turned out to be the easy part.
The hard part, in two numbers. With both models generating at once, the fast model drops from 77.5 to 28.2 tokens per second, and the box produces less total work than the fast model produces alone. And on 2026-08-23 the driver model died with exit code 0, which systemd read as a clean stop, so the fleet sat dark for thirteen hours until a failed morning brief told me. One box, two models works. It also comes with a class of failures the spec sheet does not mention, and this article is the bill.
The stack
Everything below runs on one NVIDIA DGX Spark (GB10, the ARM/Grace desktop box, 128 GB unified memory), with both models resident simultaneously under vLLM behind OpenAI-compatible endpoints:
- Qwen3.8-27B, NVFP4-quantized, served at 262K context: the careful model, our “eyes.” Slow (~11 tok/s decode), strongest local review signal.
- Nemotron-3.5-Lightning-30B-A3B, also NVFP4: the fast model, our “hands.” ~7× the decode speed, near-perfect structured output and tool-calling.
- semgrep as the deterministic floor, with repo-specific rules for our known historical bug classes.
- Claude Sonnet over the API as the eval judge (4.6 for the measurements in this article, Sonnet 5 from mid-September), used only to adjudicate eval results, at cents per decision.
- CodeRabbit as the paid ceiling: it reviews every PR after the local gate has had its shot.
- The gate itself is a small Python CLI that assembles the diff, fans out finder seeds, and runs a verifier pass. Thin plumbing.
The pair has been the configuration since 2026-08-19, on vLLM 0.27.1. The May and June findings below were taken on the single drivers before it, on vLLM 0.21.0, and each one is stamped.
Act 1: Memory tetris, and the wall that wasn’t there
The log opens on 2026-05-20 with three vLLM units sharing the pool, 101 of 121 GB in use, swap already touched, and a 4B utility model serving 24K of context to an agent framework that demanded 64K before it would compress a conversation. That bug is why the log exists. On 2026-05-24 we collapsed everything onto one model, Qwen3.6-35B-A3B-NVFP4 (vLLM 0.21.0) at 80% of memory, 65,536 context, speculative decoding on, vision served from the same unit. It benchmarked at 855 tokens per second aggregate across 96 concurrent streams.
Then the audit fleet started dying at roughly 57K of accumulated context. On 2026-05-27 we blamed the quantization’s 4K calibration, called it a quality cliff, and swapped drivers to Qwen3-Next-80B-A3B-Instruct-NVFP4 at 60% of memory and 131,072 context. The wall was real. The diagnosis was not. On 2026-08-19 we re-read the May bench: the 60K prompt had returned HTTP 400 because prompt plus system prompt plus requested output exceeded 65,536. A config ceiling. The same 64,307-token prompt ran clean on the 80B at 262K. We had changed models over a flag, and “quality cliff at 57K” lived in our notes for three months as a fact.
Act 2: The memory dial and the context dial are different knobs
The belief I had to unlearn: a larger --max-model-len reserves more
memory. vLLM sizes its KV pool at boot from --gpu-memory-utilization and
hands pages out per token; max-model-len is a per-request ceiling. On
2026-06-07 we raised the 80B from 131K to 262K and the pool stayed at
20.8 GiB, 1,722,225 tokens. A full 256K request takes 3.2 GiB of it, so
the raise moved the ceiling on simultaneous full-length requests from
about 13 to about 6.5 and cost no RAM.
“Costs nothing” survived two months and failed the next measurement. On 2026-08-20 we raised the current pair from 131,072 to 262,144 and watched the pools across the restart (vLLM 0.27.1):
| engine | util | KV pool before → after | Δ | full-length concurrency |
|---|---|---|---|---|
| Nemotron-3.5-Lightning-30B-A3B NVFP4 | 0.25 | 9.12 → 8.26 GiB | −2.7% | 21.2× → 10.31× |
| Qwen3.8-27B NVFP4 | 0.45 | 30.33 → 26.8 GiB | −10.0% | 7.3× → 3.27× |
vLLM also reserves non-KV workspace that scales with the window, and takes it out of the budget before sizing the pool. Cheap, not free. The log entry originally said “allocates nothing”; the correction sits next to it, dated, because the next person would have trusted the original.
The second lesson decides how you split the box. KV per gigabyte is architectural: Nemotron’s Mamba layers carry constant-size state instead of per-token KV, so it gets about 327K tokens of cache per GiB against Qwen3.8’s roughly 32K. Split memory evenly, 30% each, and you get this (2026-08-19):
| util | KV pool | concurrency at 131K | |
|---|---|---|---|
| Nemotron | 0.30 | 4,372,707 tok | 33.4× |
| Qwen3.8 | 0.30 | 221,466 tok | 1.69× |
Same budget, ten times apart, with the long-context model starved and the tool driver holding twenty times the concurrency it can use. We rebalanced to 0.38/0.22, then to 0.45/0.25 once retiring the 80B and the vision sidecar freed about 90 GiB, and Qwen3.8’s pool went 221,466 → 613,175 → 953,661 tokens as memory came back. Steady state with the pair under systemd: 100 GiB used, 20 free, a peak of 108 of 121 during the paired bring-up. I have not added anything since.
Act 3: Two KV pools, one GPU
Co-residency costs nothing in memory. Each engine pre-allocates its pool and never touches the other’s. Compute is the story a buyer needs. The GB10 reports no MIG partitioning and no MPS daemon runs, so the driver time-slices between the two CUDA contexts and kernels from the two processes never execute in the same instant. Same prompt to both, measured 2026-08-19:
| solo (neighbour idle) | both generating | delta | |
|---|---|---|---|
| Nemotron | 77.5 tok/s | 28.2 tok/s | −64% |
| Qwen3.8 | 11.4 tok/s | 11.3 tok/s | flat |
The penalty lands on the wrong model. Qwen3.8’s dense decode issues long, bandwidth-saturating kernel sequences, and Nemotron’s short MoE steps wait behind them. Under contention the box makes about 40 tokens per second combined, against 77.5 for Nemotron alone: simultaneous load yields less total work than leaving the second model off. An earlier reading in the log said “no co-residency penalty”; it had been taken one model at a time with the other idle. The pairing is sound when the two alternate, an agent turn and then a coding turn, which is the real workload. It is not two GPUs. If your fleet needs both models busy in the same second, the second model costs the first one two-thirds of its speed. Why the roles split the way they do, eyes versus hands and thinking per role, is the routing article of this series.
Act 4: Speculative decoding, measured and then not shipped
Multi-token prediction (MTP) is vLLM’s speculative decoding for checkpoints that ship a draft head, and its history on this box is a version story. On the 80B in June it crashed at weight load, a vLLM loader bug on 0.21.0, and was parked (2026-06-09), so the driver ran without it for months for version reasons. On Qwen3.8-27B-NVFP4 under vLLM 0.27.1 it works: 2.13× faster on the 18-packet coding suite at k=3 with identical accuracy, and about 6% slower on long-document reads, measured 2026-08-19 with a control arm that differed by exactly one launch flag. The Qwen3.8 article carries the full table and the acceptance-rate math; the serving fact belongs here.
The deployed Qwen3.8 unit runs without it. MTP needs --language-model-only,
which disables the vision tower, and the unit has to answer to the vision
alias because the 4B vision sidecar was retired into it. Vision won. The
2.13× belongs to a launch we do not run, available the day a second,
language-only Qwen3.8 fits on the box. On one box a feature flag and a
served alias can be mutually exclusive, and you pick.
Act 5: The reliability class
The failures on this box sort into four shapes. All four came from the unified memory pool, or from systemd believing something that was false.
The compile storm. vLLM’s first start on a cold cache JIT-compiles
kernels, and ninja defaults its job count to the core count. On 2026-06-07,
after a system update reset the cache, the 80B loaded its weights and then
spawned about 21 compilers at roughly 3.3 GiB of host RAM each. The pool hit
121 GiB, the kernel killed the desktop session, and Restart=on-failure
re-ran the same storm every six minutes; it peaked at 124 GiB even with the
other two units stopped. Capping MAX_JOBS=4 brought the peak to 74 GiB.
On 2026-08-18 the identical incident happened again, because the caps lived
as Environment= lines inside the units and a hand-run vllm serve from a
fresh venv inherits none of them; the box was unreachable for about
nineteen minutes. The blowup was compiler processes in host RAM, and on
unified memory there is no second pool to spill into. The fix that held is
a checked-in launch script that exports the caps and refuses to start
beside a running engine.
The boot chain with a hole. After= only orders units that are in the
same boot transaction. The vision unit was disabled (running since June
because nothing had rebooted), so on the 2026-08-18 firmware reboot the
embedding unit had no live ordering edge and started in the same second as
the 80B. It survived because the compile caches were warm. A driver bump an
hour later invalidated them; two cold compile storms do not fit in 128 GB.
The readiness gate that lied. Each unit holds itself in activating
with a helper that polls /v1/models, so the chain waits for serving and
not for a forked process. The helper always exited 0. When the first
Nemotron start under systemd died at engine init (a compiler on PATH in a
shell and absent under systemd, so the same command line works by hand and
fails as a unit), the unit sat in activating for the full timeout, then
reported success for a model that never served and released every
dependent into the overlap the chain existed to prevent. Fixed 2026-08-20:
a dead main process exits 1 within seconds, a timeout exits 1, budgets sit
at 1,080 seconds against warm starts of about 200 seconds for Nemotron and
335 for Qwen3.8, and StartLimitBurst=3 per hour bounds any loop. A
readiness helper must report on itself.
The crash that exited 0. On 2026-08-23 at 16:56 the Nemotron engine hit
CUDA error: an illegal instruction was encountered in its Mamba-2 path, a
known open race on this GPU (vLLM issue 37431). Qwen3.8 and the embedding
model on the same GPU ran through it untouched. The engine process died
fatally; the API server then shut down cleanly and the unit logged
Deactivated successfully, so Restart=on-failure never fired. The service
stayed down until the 06:00 morning brief failed on a connection error, and
I started it by hand at 06:56. The load shape before it was new: a
knowledge-base ingest had held the driver in twelve-way sustained
long-decode batches for about 34 hours. The fix is a one-line drop-in,
Restart=always, with the burst limit still bounding a genuine loop.
(Config log entry dated 2026-08-24, backfilled from the drop-in, the unit
journal, and the Hermes runtime notes.)
The restart budget
The operator’s recurring cost is the cold start, because it sets every timeout in the stack and it is disk-bound: vLLM disables its weight prefetch here because the checkpoint exceeds 90% of what it counts as available RAM, so weights stream off EXT4 at 25 to 30 seconds per shard.
| model | cold start | restart, caches warm | date |
|---|---|---|---|
| Qwen3-Coder-30B FP8 (57 GB) | 8 to 10 min | ~30 s | 2026-05-19 |
| Qwen3.6-35B NVFP4 | ~4 min | sub-minute | 2026-05-24 |
| Qwen3-Next-80B NVFP4 (11 shards) | 5m46s, 6m5s | ~6 min | 2026-05-27, 06-07 |
| Nemotron-3-Super-120B NVFP4 | ~9 min | 2026-06-08 | |
| Qwen3.8-27B NVFP4 | 210 s to ~335 s | 2026-08-20 | |
| Nemotron-3.5-Lightning-30B NVFP4 | ~200 s | 2026-08-20 |
Two engines cold-compiling at once do not fit, so the units serialize and a full cold boot of the pair is the sum, inside a readiness budget of 1,080 seconds each. Every cron job on the box has to respect that after a reboot.
One desk box against a cluster
For a small team’s agent fleet whose work alternates between a tool loop and
a coding or vision escalation: yes, on the evidence above, with 20 GiB of
headroom and nothing else added. Two models resident, 262K context on both,
cache pools sized to what each architecture can use, zero dollars per token.
What the operator signs up for is the list this article is made of: compile
caps on every launch path including the interactive one, serialized cold
starts behind honest readiness gates, Restart=always because a fatal error
can exit 0, a five-minute-per-model restart budget after any reboot, and one
GPU’s worth of compute that the second model cannot borrow. I also signed up
for reading vLLM’s request contract after it cost me a benchmark: requested
output counts against the context window, and a 400 from the engine is a
capacity error, never a quality result. Thirty-six benchmark cells were
scored as model failures before that sank in.
The playbook
- Size memory by utilization, treat context as a per-request ceiling, then measure the raise anyway. 131K to 262K cost 0 RAM on the 80B in June and 10% of Qwen3.8’s KV pool in August. The pool numbers after the restart are the only ones that count.
- Split the pool by KV per gigabyte, not evenly. A Mamba hybrid and a dense-attention model differ by ten times in tokens per GiB; the memory belongs to the KV-starved half.
- Measure co-residency with both models generating. One-at-a-time measurements show memory isolation and hide the compute penalty. Ours was 64% on the fast model and a net loss in total throughput.
- Cap compile parallelism on every launch path. A cap in a unit file protects the unit. The hand-run path took the box down twice with the same fault; a script that refuses is a control, and a comment is not.
- Serialize cold starts and make the readiness gate fail. A helper that exits 0 on timeout releases dependents into the overlap it exists to prevent. A dead process or a timeout exits 1.
- Use
Restart=always, bounded by a burst limit. An engine can die fatally and hand systemd a clean exit.on-failurewatched ours stay down for thirteen hours. - Read the engine’s request contract before the benchmark. Prompt plus requested output must fit the window, and a 400 belongs in its own bucket, outside every quality numerator and denominator.
- Record the correction next to the claim. “Quality cliff at 57K” and “allocates nothing” both lived as facts until measured. The log keeps the wrong version beside the right one, dated.
This is Spark Bench: what one DGX Spark on a desk can and cannot be trusted to do, measured honestly. Next: the two models on this box did not start as a pair. A tool-chain audition put a 30B model with 3B active parameters in the driver seat, and the split between the two beats either one alone. How that routing was measured, and where it fails, is the next article. Follow for the routing story.
Appendix: the numbers behind the story
Everything below is copied from the dated change log
(docs/dgx/vllm-config-log.md, 3,062 lines, 2026-05-20 to 2026-09-14), the
two checked-in unit files, the readiness helper, the 2026-08-19 eval
artifacts, and the 2026-06-11 llama-benchy record. Where a number was
corrected later, both values appear with their dates. Nothing here was
re-measured for the article.
The served pair, as configured
One NVIDIA DGX Spark: GB10 (Blackwell, SM_121), 128 GB unified memory (121 GiB visible to the OS), no MIG partitioning, no MPS daemon. Weights on EXT4. Both engines run from the same vLLM 0.27.1 venv; the three retired units ran vLLM 0.21.0 from an older venv, and only the embedding unit still does.
| flag | vllm-nemotron (:8012) | vllm-q38 (:8011) |
|---|---|---|
| checkpoint | Nemotron-3.5-Lightning-30B-A3B-NVFP4 | Qwen3.8-27B-NVFP4 (llm-compressor format) |
--quantization |
modelopt | compressed-tensors (modelopt fails engine start) |
--gpu-memory-utilization |
0.25 | 0.45 |
--max-model-len |
262,144 (native 1,048,576) | 262,144 (native maximum) |
--max-num-seqs |
128 | 128 |
--served-model-name |
nemotron | q38 and vision |
| tool parser | qwen3_xml | qwen3_xml |
| reasoning parser | nemotron_v3 | none (callers send enable_thinking: false) |
--generation-config |
default | vllm (the checkpoint ships thinking-mode sampling as server defaults) |
| speculative decoding | none | none (MTP needs --language-model-only, which drops the vision path) |
| readiness gate | wait-ready 8012, 1,080 s | wait-ready 8011, 1,080 s |
TimeoutStartSec |
1,200 s | 1,200 s |
StartLimitIntervalSec / Burst |
3,600 s / 3 | 3,600 s / 3 |
Restart= |
always (drop-in, 2026-08-24; base unit says on-failure) | on-failure |
| ordering | after network-online | after vllm-nemotron (no Requires=) |
Both units export MAX_JOBS=4, NVCC_THREADS=1,
TORCHINDUCTOR_COMPILE_THREADS=4, put /usr/local/cuda/bin first on PATH
(FlashInfer needs nvcc at engine init; without it the bmm_fp8 op never
registers and any cached compile artifact that references it dies), and set
CUDA_HOME. --max-num-seqs 128 on both is not a memory setting: on a Mamba
hybrid every decode sequence needs one Mamba state block, and the default of
256 exceeded the pool’s block count, so vLLM refused CUDA-graph capture with
max_num_seqs (256) exceeds available Mamba cache blocks (147). The
embedding unit (bge-m3, :8004) carries a drop-in ordering it after both
drivers, ordering only, no Requires=.
KV pool ledger
Every pool figure vLLM printed at boot, in order. Tokens per GiB is pool divided by KV memory; it is the architectural constant the split is built on.
| date | engine | util | KV memory | KV pool (tokens) | concurrency at window | note |
|---|---|---|---|---|---|---|
| 2026-05-24 | Qwen3.6-35B-A3B NVFP4 | 0.80 | ~70 GB budget | ~350K | ~5× at 65,536 | single driver, MTP k=3, vision in-unit |
| 2026-06-07 | Qwen3-Next-80B NVFP4 | 0.60 | 20.8 GiB | 1,722,225 | 13.1× at 131,072 | before the 262K raise |
| 2026-06-09 | Qwen3-Next-80B NVFP4 | 0.60 | 24.75 GiB | 2,106,551 | 8.04× at 262,144 | after restart, max-num-seqs 16 |
| 2026-06-12 | Qwen3-Next-80B NVFP4 | 0.60 | 24.64 GiB | 2,097,152 | 8.00× at 262,144 | max-num-seqs 32; pool unchanged by the knob |
| 2026-08-19 | Nemotron | 0.30 | 14.36 GiB | 4,372,707 | 33.4× at 131,072 | even split, audition |
| 2026-08-19 | Qwen3.8 | 0.30 | 7.05 GiB | 221,466 | 1.69× at 131,072 | even split, audition |
| 2026-08-19 | Qwen3.8 | 0.38 | 19.49 GiB | 613,175 | 4.68× at 131,072 | rebalanced |
| 2026-08-19 | Nemotron | 0.22 | 5.67 GiB | 1,725,781 | 13.2× at 131,072 | rebalanced |
| 2026-08-19 | Nemotron | 0.25 | 9.12 GiB | 2,777,998 | 21.2× at 131,072 | systemd, after retirements |
| 2026-08-19 | Qwen3.8 | 0.45 | 30.33 GiB | 953,661 | 7.28× at 131,072 | systemd, after retirements |
| 2026-08-20 | Nemotron | 0.25 | 8.26 GiB | 2,703,604 | 10.31× at 262,144 | window raised; pool −2.7% |
| 2026-08-20 | Qwen3.8 | 0.45 | 26.8 GiB | 858,483 | 3.27× at 262,144 | window raised; pool −10.0% |
Tokens per GiB from the systemd rows: Nemotron about 305K (2,777,998 / 9.12), Qwen3.8 about 31K (953,661 / 30.33). The log quotes both “~304K vs ~31K” and “~327K vs ~32K” depending on which row it divides; the ratio is ten times either way. Per-token KV from config.json with fp8 KV, recorded 2026-06-07: Qwen3-Next-80B 12 KB (12 of 48 layers full attention), Qwen3.6-35B 10 KB (10 of 40), Qwen3-32B dense 128 KB (native context 40,960, disqualified).
The 2026-08-20 raise, prediction against restart:
| engine | predicted concurrency (“allocates nothing”) | measured after restart |
|---|---|---|
| Nemotron | 10.6× | 10.31× |
| Qwen3.8 | 3.6× | 3.27× |
Retrieval at the served window
Needle probe: an un-inferable fact planted at 10, 50, and 90 percent depth in
a haystack of real repository source, asked for verbatim, scored by exact
match in code. Haystack length measured with the served model’s own tokenizer
via /tokenize, because tiktoken over-counts about 8 percent here (a 20K-token
error at a 240K target, enough to produce a 400 that reads as a quality
cliff).
Qwen3.8, 2026-08-19, greedy, no decoys, both launch arms:
| arm | 64K (in-tok) | 128K (in-tok) | seconds per case at 64K / 128K |
|---|---|---|---|
| no MTP | 3/3 (64,701) | 3/3 (129,314) | 39.1 / 106.9 |
| MTP k=3 | 3/3 (64,314) | 3/3 (128,712) | 41.7 / 113.6 |
The 240K arm (242,379-token haystack) was abandoned when one case passed eight minutes; the engine accepted the prompt, which is the evidence that the hand-launched engines were already above 131,072.
Both drivers, 2026-08-20, greedy, two decoys sharing the needle’s anchor phrase, measured before the raise:
| engine | 128K, depths 10/50/90 | seconds per case |
|---|---|---|
| Nemotron | HIT HIT HIT | ~29 |
| Qwen3.8 | HIT HIT HIT | ~108 |
After the raise, 256K with two decoys, one haystack composition:
| engine | tokens | 10% | 50% | 90% | seconds per case |
|---|---|---|---|---|---|
| Nemotron | 256,258 | HIT | HIT | DECOY | 78.6 |
| Qwen3.8 | 258,431 | HIT | HIT | HIT | 330 |
Nemotron bisect on the same haystack (greedy, two decoys, four depths):
| tokens | 10% | 50% | 75% | 90% |
|---|---|---|---|---|
| 161,418 | HIT | DECOY | HIT | DECOY |
| 194,019 | HIT | DECOY | HIT | HIT |
| 225,032 | HIT | HIT | DECOY | HIT |
Control, same 256K length, different corpus: 257,167 tokens, HIT at all four depths. The reading “Nemotron degrades at depth by 256K” was withdrawn on that control; what survives is that Nemotron sometimes prefers a decoy in long contexts, at roughly a third of cells, depending on where the decoys land. Depth 10 percent was clean in every run.
One more cell, and the one that changed the plan: at the production sampling
point (temperature 1.0, top_p 0.95) on a 16,327-token haystack at 50 percent
depth, Nemotron retrieved 3 of 5, with a decoy on the misses. Greedy is clean
there. n=5 on one cell, so it drives nothing yet; it is why the agents’
context_length stayed at 126,976 after the serving window doubled.
Decode and throughput
Four Nemotron decode figures appear in the log. They are four setups, and the spread is about 3 percent:
| figure | setup |
|---|---|
| 76.7 tok/s | audition, hand-launched at util 0.30, Qwen3.8 audition engine resident and idle |
| 77.5 tok/s | solo, neighbour idle (the co-residency table’s baseline) |
| 78.8 tok/s | systemd at util 0.25, neighbour idle, EOS-terminated generation |
| 79.2 tok/s | same, generation pinned to 512 tokens via min_tokens and ignore_eos, n=3 (2026-08-20) |
Qwen3.8 measured 11.3 tok/s in both the 2026-08-19 and the pinned 2026-08-20 runs (n=3). The pinned re-measure exists because an early EOS shortens the decode and pays a larger share of fixed prefill, which biased terse models downward; the 7× ratio did not move.
Co-residency, 2026-08-19, same prompt to both engines, one run each:
| engine | solo | both generating | delta |
|---|---|---|---|
| Nemotron | 77.5 tok/s | 28.2 tok/s | −64% |
| Qwen3.8 | 11.4 tok/s | 11.3 tok/s (minimum 7.2) | flat at the median |
Combined under contention about 40 tok/s. The earlier log line “no co-residency penalty (76.7 → 78.8)” compared two neighbour-idle readings.
Long-context sweep, 2026-08-19, 512 output tokens, each model at its own
card-recommended Instruct sampling (temperature 0.7, top_p 0.8, top_k 20;
Qwen3.8 adds presence_penalty 1.5), enable_thinking: false, hand-launched
engines:
| prompt | input tokens (80B / Qwen3.8) | Qwen3-Next-80B | Qwen3.8-27B | penalty |
|---|---|---|---|---|
| longctx-4k | 4,356 / 4,687 | 13.9 s, 36.9 tok/s | 49.8 s, 10.3 tok/s | 3.6× |
| longctx-16k | 20,909 / 22,547 | 17.9 s, 28.6 tok/s | 60.0 s, 8.5 tok/s | 3.4× |
| longctx-32k | 36,941 / 39,866 | 18.7 s, 27.4 tok/s | 70.9 s, 7.2 tok/s | 3.8× |
| longctx-60k | 64,307 / 69,744 | 24.9 s, 20.6 tok/s | 95.4 s, 5.4 tok/s | 3.8× |
Qwen3.8’s tokenizer (vocabulary 248,320) emits 7.6 to 8.5 percent more tokens for identical text, which is the second column’s spread. The 64,307-token prompt is the one that returned HTTP 400 on the 35B in May.
Qwen3-Next-80B under llama-benchy 0.3.8, 2026-06-11, vLLM 0.21.0, util 0.60, max-num-seqs 16, prompt 2,048 tokens, decode 512 (exact), three runs per cell:
| concurrency | aggregate decode tok/s | per-stream tok/s | aggregate prefill tok/s |
|---|---|---|---|
| 1 | 37.9 | 37.9 | 3,884 |
| 2 | 69.9 | 34.9 | 4,569 |
| 4 | 112.8 | 28.4 | 4,434 |
| 8 | 173.4 | 22.0 | 5,271 |
| 16 | 243.5 | 16.4 | 5,346 |
| context depth | decode tok/s | time to first token |
|---|---|---|
| 0 | 37.4 | 0.5 s |
| 4,096 | 37.0 | 1.3 s |
| 16,384 | 36.0 | 4.0 s |
| 65,536 | 32.8 | 20.0 s |
The same engine at max-num-seqs 32, 2026-06-12, depth 0, all three units up:
| concurrency | aggregate decode tok/s (± sd) | per-stream | aggregate prefill | time to first response |
|---|---|---|---|---|
| 8 | 174.3 ± 4.0 | 22.2 | 4,954 | 3.1 s |
| 16 | 235.8 ± 4.0 | 15.9 | 5,272 | 4.9 s |
| 24 | 281.0 ± 3.7 | 12.9 | 5,315 | 6.5 s |
| 32 | 315.4 ± 4.3 | 11.0 | 5,194 | 8.2 s |
Settled at 32; 48 and 64 were not ratcheted because the KV pool caps full-window streams at 8 regardless and time to first response was climbing linearly. Running deep context and high concurrency in one benchy invocation deadlocked the engine on 2026-06-11 (prefill-dominated 120:1, requests stuck running, restart required); the standing rule is depth 0 or concurrency 1, never both.
Qwen3.6-35B-A3B NVFP4, 2026-05-24, vLLM 0.21.0, util 0.80, MTP k=3,
six-prompt orchestrator set, enable_thinking: false:
| concurrency | aggregate tok/s | p50 latency | p95 latency |
|---|---|---|---|
| 1 | 48 | 2.8 s | 6.7 s |
| 8 | 268 | 2.4 s | 13.4 s |
| 32 | 579 | 4.1 s | 23.3 s |
| 48 | 701 | 5.4 s | 29.4 s |
| 64 | 753 | 6.6 s | 34.9 s |
| 96 | 855 | 9.2 s | 46.3 s |
| 128 | 769 | 11.2 s | 50.6 s |
MTP contributed 1.4 to 1.6× over no-MTP on that model; thinking on at c=32 gave 5.4× worse per-call latency for the same aggregate (1.05×) and the same final answers.
Speculative decoding on Qwen3.8 (audition only, not deployed)
Launch flag --speculative-config '{"method":"mtp","num_speculative_tokens":k}' --language-model-only, engine architecture Qwen3_5MTP, draft head
model_mtp.safetensors (15 tensors, one layer). Same four prompts, 512
output tokens, identical sampling:
| prompt | no MTP | k=1 | k=2 | k=3 | best |
|---|---|---|---|---|---|
| longctx-4k | 49.8 s | 30.3 s | 29.0 s | 27.4 s | k=3, 1.82× |
| longctx-16k | 60.0 s | 42.1 s | 35.9 s | 33.1 s | k=3, 1.81× |
| longctx-32k | 70.9 s | 54.1 s | 47.4 s | 46.4 s | k=3, 1.53× |
| longctx-60k | 95.4 s | 79.7 s | 72.2 s | 73.8 s | k=2, 1.32× |
Conditional acceptance per draft position:
| depth | pos 0 | pos 1 | pos 2 | extra tokens per pass | tokens per forward pass |
|---|---|---|---|---|---|
| k=1 | 78.3% | 0.78 | 1.78 | ||
| k=2 | 80.9% | 74.8% | 1.41 | 2.41 | |
| k=3 | 77.3% | 73.8% | 71.3% | 1.75 | 2.75 |
The k=3 closed form (1 + .773 + .773×.738 + .773×.738×.713 = 2.75) matches the
measured 1,304 tokens over 745 passes. Enabling speculation forces
max_num_scheduled_tokens to 2,048, so a 64K prompt prefills in about 32
chunks instead of 8; that is the long-context tax. min_p and logit_bias
silently stop working under speculation.
The deciding measurement, control arm differing by exactly the one launch flag, prefix caching off in both:
| workload | no MTP | MTP k=3 | delta |
|---|---|---|---|
| code generation, 18 packets (short prompt, long output) | 35m59s | 16m53s | 2.13× faster |
| long-document read at 64K | 39.1 s | 41.7 s | 6.6% slower |
| long-document read at 128K | 106.9 s | 113.6 s | 6.3% slower |
| coding accuracy (Opus judge) | 18/18 | 18/18 | identical |
| retrieval fidelity | 6/6 | 6/6 | identical |
The 18-packet suite by rung, temperature 0.2, cost.local_wall_s summed
(model generation only, judge time excluded):
| rung | n | first pass, no MTP → k=3 | recovered | mean wall s, no MTP → k=3 |
|---|---|---|---|---|
| r0 | 3 | 0.667 → 1.0 | 1.0 → 1.0 | 46.2 → 25.9 |
| r1 | 5 | 1.0 → 1.0 | 1.0 → 1.0 | 114.6 → 78.5 |
| r2 | 5 | 1.0 → 1.0 | 1.0 → 1.0 | 78.6 → 54.7 |
| r3 | 5 | 0.8 → 1.0 | 1.0 → 1.0 | 210.8 → 53.8 |
The first-pass change is run-to-run variation, not MTP improving quality; speculation with rejection sampling is distribution-preserving, and the figure that matters is that recovered stayed 1.0 on every rung. Accuracy was measured at k=3 and no-MTP only; the k=1 and k=2 rows are timing and acceptance. Every MTP number is single-stream. Prefix caching, measured the same day: hit rate tracks complete 1,568-token blocks exactly (0 percent below 1,568 tokens, 63.6 percent at 2,467, 95.8 percent at 8,187, 95.7 percent at 32,757), costs about 7 percent of the KV pool (1,232,536 → 1,144,996 tokens), and the 6.86× wall-clock speedup at 32,757 tokens is a floor because the probe’s ascending sweep pre-warmed later targets; re-run before quoting it.
Incident ledger
| date | unit | trigger | peak or duration | fix that held |
|---|---|---|---|---|
| 2026-06-07 15:10 | vllm-qwen3-next (80B) | system update reset the JIT cache; three units booted concurrently; ~21 cicc/nvcc at ~3.3 GiB each | 121 GiB, global OOM; 124 GiB solo; restart loop every ~6 min | MAX_JOBS=4, NVCC_THREADS=1, TORCHINDUCTOR_COMPILE_THREADS=4 (peak 74 GiB, max cicc 4); After= chain plus readiness gate on all three units the same day |
| 2026-06-11 | vllm-qwen3-next | benchy sweep crossing depth 65,536 with concurrency 16 | engine deadlock, HTTP 000; ~6 min restart; embed lost the concurrent-warmup race with CUDA OOM once | bench rule: depth 0 or concurrency 1 |
| 2026-08-18 08:51 to 09:10 | hand-run vllm serve (8B NVFP4 smoke test, fresh venv, util 0.15) | cold FlashInfer cache, 47 CUTLASS FP4 artifacts compiled at MAX_JOBS=20 | ~19 min unreachable; load average 155 with every process under 4% CPU; swap 15.1 of 16 GiB | caps exported on every interactive launch; checked-in launch script with lock file and co-residency preflight |
| 2026-08-18 12:00 | vllm-embed and vllm-qwen3-next | first real reboot since June; both After= edges of embed dead (one unit disabled, one retired) |
both started at 12:00:31, identical second; survived on warm caches only | edge added to a live unit; stale drop-in removed; vision re-enabled |
| 2026-08-19 | vllm-nemotron first systemd start | /usr/local/cuda/bin absent from unit PATH; bmm_fp8 unregistered; cached artifact referenced it |
unit sat in activating for the full 900 s budget then reported success | PATH and CUDA_HOME in the unit; helper exits 1 on dead $MAINPID |
| 2026-08-23 16:56 | vllm-nemotron | CUDA error: an illegal instruction was encountered, Mamba-2 path (vllm-project/vllm#37431), 32 running requests, after ~34 h of 12-way long-decode batches |
API server exited 0, unit logged Deactivated successfully; down until the 06:00 brief failed; hand-started 06:56 | Restart=always drop-in, 2026-08-24 |
Symptom profile of the 2026-08-18 storm, for recognising the next one: LAN ping 0 percent loss but jitter 2.7 to 90 ms; TCP handshake on :22 succeeds from the kernel backlog; SSH banner takes 136 s; tailscale ping times out; memory pressure (PSI) avg300 82.28, CPU 76.28; the OOM killer resolved it unaided.
Readiness helper (vllm-wait-ready.sh) exit semantics:
| condition | before 2026-08-20 | after |
|---|---|---|
endpoint answers /v1/models |
exit 0 | exit 0 |
$MAINPID gone |
exit 0 after the full timeout | exit 1 immediately |
| timeout, process alive | exit 0 | exit 1; budget raised 420 s → 1,080 s |
The helper counts wall clock, not sleeps: a poll iteration can cost about 10 s
(a 5 s curl against a socket that accepts and stalls, plus the sleep), so
counting sleeps let the real budget reach twice the nominal one, past
TimeoutStartSec. The embedding unit shares the helper (8004 270), so the
change reached it too; on 2026-08-18 that unit had hit 270 s with a dead
engine and proceeded anyway.
Cold and warm starts
| model | checkpoint size | cold start | warm restart | date | source |
|---|---|---|---|---|---|
| Qwen3-Coder-30B-A3B (unit id ends -FP8) | 56.87 GiB, 16 shards | 8 to 10 min | ~30 s; 2 to 4 min if flags force partial recompile | 2026-05-19 | bring-up spec |
| Hermes-4.3-36B-FP8 | 35.16 GiB | 321 s | 2026-05-22 | config log | |
| Qwen3.6-35B-A3B NVFP4 | ~4 min (weights, graphs, MTP compile) | sub-minute when shards stay in page cache | 2026-05-24 | config log | |
| Llama-3.3-70B (repo D’s unit) | ~7 min | 2026-05-25 | config log | ||
| Qwen3-Next-80B NVFP4 | 11 shards, ~31 s each | 5m46s, 6m1s | ~6 min (06-09); 440 s at 84 GB peak, zero cicc (06-12) | 2026-05-27, 06-09, 06-12 | config log |
| Nemotron-3-Super-120B NVFP4 | ~9 min | 2026-06-08 | config log | ||
| Qwen3.8-27B NVFP4 | 21.81 GiB weights in 128 s | ~4 min hand-launched, peak 80 GB | 210 s (08-20 restart); ~335 s warm under systemd | 2026-08-18, 08-20 | config log |
| Nemotron-3.5-Lightning-30B NVFP4 | ~200 s warm under systemd | 2026-08-20 | config log |
vLLM disables its weight prefetch on this box because the checkpoint exceeds 90 percent of what it counts as available host RAM (it reserves its GPU share before measuring), so weight loading is disk-bound on EXT4 at about 25 to 30 seconds per shard regardless of page-cache state.
Memory over time
| date | state | used / available (GiB) |
|---|---|---|
| 2026-05-20 | three units (coder 0.50, utility 0.15, vision), swap 3.6 GB touched | 101 / ~20 |
| 2026-05-27 | Qwen3-Next-80B at 0.60, idle after warm | 80 / 40 |
| 2026-06-07 | 80B, vision, embed, after the capped restart | 106 / 15 |
| 2026-06-08 | Nemotron-120B at 0.80 plus embed | 110 / 11 (117 / 4 at 0.85) |
| 2026-08-18 | 80B stopped for the smoke test | 81 free |
| 2026-08-19 | vision and 80B retired, two audition engines still up | 87 used, 34 free (from 106) |
| 2026-08-19 | paired bring-up peak, then steady, audition utils | 108 peak; 108 / 13 |
| 2026-08-19 | pair under systemd at 0.25 / 0.45 | 100 / 20 |
The units also carry cgroup caps from the June hardening: MemoryHigh=105G
as the soft cap and, from 2026-06-12, MemoryMax=112G so a runaway compile
kills the unit instead of the box. MemoryHigh alone throttles and reclaims;
it does not stop a cgroup from driving the host into global OOM, which is how
the 2026-08-18 interactive launch escaped every protection the units had.
The request contract, in three lines
Measured against vLLM 0.27.1 on the Nemotron driver at a 262,144 window.
max_tokens is spent from the same window as the prompt, and the engine
rejects the sum: a 261,380-token diff is unreviewable at any max_tokens
above 764. That 400 arrives as a client exception; in the 2026-08-20 reviewer
benchmark the harness recorded it as a format failure on 36 cells, one entire
pull request across every arm. finish_reason: "length" returns HTTP 200
with a truncated body, and reasoning tokens are spent from the same
max_tokens as the answer. /tokenize is a pre-template lower bound; the
exact count comes back in usage.prompt_tokens.
What stays private
The raw run JSONs behind the coding and MTP timings live in gitignored
directories on the box (evals/local_executor/runs/,
scripts/bench/results/); the per-packet verdict summaries next to them
carry no wall-clock field, so the durable record is the change log entry
quoted above. The needle haystacks are built from this repository’s own
source and are not published. The Hermes profile configurations that consume
these endpoints are not published. Unit files, the launch script, the
readiness helper, and every aggregate on this page are the reproducible part.
Written under Spark Bench style v0.1