Spark Bench · Article 5
My DGX Spark Reads 96% Busy and Is Mostly Waiting
On the DGX Spark on my desk, nvidia-smi read the GPU 96% busy while one code-review
eval sent Qwen3.8-27B (NVFP4, vLLM 0.27.1) requests serially. Nsight Systems
showed the streaming multiprocessors issuing on 3.1% of cycles. I measured what
concurrent streams buy on Qwen3.8 and on Nemotron-3.5-Lightning-30B-A3B (NVFP4)
beside it, and what they cost the eval next door. Eight review cells took 4.7 times
less wall clock at eight workers. A second Qwen3.8 job cost the running eval 5 to 9%,
and a Nemotron job on the other engine cost it 54%. Batch Qwen3.8 by the work: eight
streams for generation-heavy cells, four for ceiling-bound review, two for long
prompts. Never batch the other engine while a ceiling-bound eval runs.
The stack
One NVIDIA DGX Spark, GB10, 128 GB unified memory. Qwen3.8-27B NVFP4 and Nemotron-3.5-Lightning-30B-A3B NVFP4 resident side by side, one vLLM 0.27.1 engine each. No judge model: every number here is a counter, a timer, a hardware sample, or a token count from a record. My Claude Code sessions and I wrote the probes and ran them.
Act 1: The busy gauge reports a resident kernel
On 2026-09-17 the golden-set review eval had been running serially against Qwen3.8 for
most of a day, and every check read the same: utilization.gpu at 96%, power.draw
around 37 W. I wanted to know whether the box was saturated or whether I was waiting
on a queue of one.
utilization.gpu is the fraction of sample periods in which any kernel was resident
on the GPU. During decode a kernel is always resident, so the field pins high whether
the chip is straining or idling between memory fetches. It cannot tell the two apart.
Nsight Systems can. It identifies the GB10 as a Blackwell GB20B and samples per-cycle
counters at 10 kHz. A ten-second capture during the eval’s single-stream decode read
GR Active at 99.9%, which is the graphics engine reporting a resident kernel and the
same thing nvidia-smi reports. SM Issue, the rate at which the multiprocessors issue
instructions, was 3.1%. The SMs stalled on nearly every cycle, the signature of decode
waiting on memory.
Then I captured again inside sustained loads of 8 and 32 streams. SM Issue read 2.8% and 3.6% while throughput rose more than eightfold. Batching makes one read of the weights serve more tokens, and the SMs do no more work per token. The counter that should climb, memory bandwidth, is not in the one metric set this GPU accepts, so the ceiling in the next act is read from the throughput curve rather than from a counter.
Act 2: Eight streams cost Qwen3.8 a tenth; Nemotron tops out at eight
A probe fires N identical completions at one engine, each pinned to exactly 512 output tokens behind a short prompt, and reads the engine’s own generation counter before and after. Decode on a 27B dense model reads the full weight set once per step, whatever the batch, so if the chip has room, N streams should cost little more than one.
| streams | Qwen3.8 per-stream tok/s | Qwen3.8 aggregate tok/s | power W |
|---|---|---|---|
| 1 | 11.1 | 11.1 | 34 |
| 8 | 10.1 | 80.5 | 37 |
| 16 | 8.6 | 137.4 | 38 |
| 32 | 5.8 | 184.4 | 40 |
Eight streams bought 7.2 times the aggregate tokens and cost each stream 10%. Thirty-two streams still gained, at 48% slower per stream. Power moved a few watts across the whole range, so it is no better a load gauge than utilization.
Nemotron is a 30B mixture-of-experts with 3B active parameters and a Mamba hybrid architecture, and its curve has a different shape. On 2026-09-19, with Qwen3.8 idle, it decoded 82 tok/s alone and 3.2 times that in aggregate at 8 streams. At 16 the aggregate fell, from 262 to 181, and the probe’s wall time tripled. Power jumped to 66 W, the only time in these runs that power tracked load, and it did so while the engine was thrashing rather than batching. Eight streams is Nemotron’s ceiling.
Act 3: Real cells keep most of the gain unless the prompt is the work
A decode probe with a short prompt is the best case. A review cell reads a diff of thousands of tokens before it writes anything, and that prefill is compute-bound, so concurrent cells contend for the multiprocessors the probe left idle. I added a worker count to the eval runner and ran fixed sets of real cells at several depths.
The Qwen3.8 thinking reviewer on eight pull requests is mostly generation: 4K to 15K output tokens a cell. Overnight on 2026-09-19, on an otherwise idle engine, the set took 8,507 seconds at one worker and 1,801 at eight. Each cell decoded at a median 9.2 tok/s against 11.6 serially, a 20% cost per cell. One cell hit the per-cell ceiling at both depths, so the eight-worker wall is the ceiling itself.
Nemotron’s card reviewer is the opposite case. Its cells are 98% prompt by token count and write a few hundred tokens. Eight workers bought 1.9 times, and each cell took three and a half times as long. No worker count changes that, because eight prefills share the same compute.
Act 4: The neighbour pays 5 to 9%, or half
Every record the eval writes carries its token counts, so the serial Qwen3.8 eval that ran through all of this was its own instrument: completion tokens over elapsed seconds, per cell. Its undisturbed baseline over 113 cells was 11.51 tok/s. Then I read the cells that overlapped each load I imposed.
| load elsewhere on the box | serial eval decode tok/s | cost |
|---|---|---|
| one more Qwen3.8 stream | 10.97 | 5% |
| four more Qwen3.8 streams | 10.0 to 10.9 | 9% |
| Nemotron eval at 4 workers on the other engine | 5.25 | 54% |
Same-engine batching shares the weight read. The other engine competes for the same memory system, and the cost scales with its batch. Three of the five serial cells inside the Nemotron window died at the ceiling; one of them needed 1,072 seconds at full speed on its next seed, so at half speed it could not fit. I launched that Nemotron job without measuring first, and the three cells were re-run on a quiet engine before anything was scored.
The ceiling also bit my own review tool. Its deep pass runs three finder seeds against Qwen3.8, each allowed 32K output tokens inside a 30-minute ceiling. Three seeds sharing the engine decode at about 10.5 tok/s each, so a seed can emit about 19K tokens before the deadline. A seed that ran long always timed out, and a non-streaming call returned nothing. Since 2026-09-22 the finder streams against its own deadline and keeps every complete finding a cut-off seed emitted, marking the seed degraded instead of lost.
The decision
My rule since these runs (2026-09-17 to 09-23), applied by hand through the eval
runner’s --concurrency flag: Qwen3.8 work batches by what the cells spend their time
on. Generation-heavy cells that finish well inside the ceiling go eight at a time;
ceiling-bound review cells under about 20K prompt tokens go four; long-context cells
go two. The Nemotron lane yields while any Qwen3.8 eval with a per-cell ceiling is in
flight. The numbers that decided it: 4.7 times on real cells at eight workers, against
54% taken from the neighbour when the other engine batches.
The constraint is the per-cell ceiling. Every eval and the review tool cap a cell at 30 minutes, so anything that slows a stream converts directly into timeouts. Staying serial lost because it leaves most of the engine idle while the gauge reads busy. A flat eight for everything lost because the ceiling-bound finder paid for it: the first job I ran at eight streams, on 2026-09-22, had 20 to 37 cells per arm cut off at the ceiling, against 5 of 93 at four streams. Those arms also changed the prompt, so part of that is the prompt, but a stream running 20% slower is exactly what turns a cell that fit into one that does not. Batching Qwen3.8 to 16 or 32 lost for the same reason, harder.
Batching the Nemotron lane beside a Qwen3.8 eval lost on the neighbour’s bill in Act 4. Raising the ceiling lost too. On 2026-09-19 I doubled it to an hour on the four PRs that timed out most: five cells ran into the 32K output cap and returned nothing, one more bug was caught, and precision halved.
The cost is per-stream latency. A cell at eight workers runs 20% slower than alone, and a stream at 32 runs 48% slower. Nemotron batching is worth about 2x on prompt-heavy work and no more. The cross-engine cost is measured in one direction only, Nemotron loading Qwen3.8; what a Qwen3.8 batch costs a Nemotron job is unmeasured. The Nsight captures ran as root through a sudoers rule the 2026-09-19 security review flagged as a root path; I removed it on 2026-09-22, so re-running them means putting it back.
Queued to revisit: the reverse cross-engine direction, and every row below when either model changes.
| queue rule | evidence |
|---|---|
| Qwen3.8 generation-heavy cells batch to 8 | Act 3; neighbour cost in Act 4 |
| Ceiling-bound review cells under about 20K prompt tokens batch to 4 | cut-offs at 8 streams, Appendix E |
| Long-context cells batch to 2 | 40K to 44K-token prompts at 4 workers decoded about 30% slower, and 8 of 24 cells timed out |
| Prompt-heavy cells cap near 2x | Act 3, Nemotron card cells |
| No batched Nemotron job beside a ceiling-bound Qwen3.8 eval | Act 4 |
| Nemotron lane at most 8 streams, sustained 2 to 4 | Act 2; the August engine crash fired under sustained 12-way batches |
Read load from token counts, never from nvidia-smi |
Act 1 |
The playbook
- A gauge that never moves under load is not measuring load.
- Batch on the engine; do not batch across engines.
- Measure real cells before quoting a decode curve; prefill sets the ceiling for short output.
- Read the cost to the neighbour from the neighbour’s own records.
- Under a wall-clock ceiling, stream and keep what arrives.
- Profile with the tool that supports the part, and write down which one does not.
This is Spark Bench: what one DGX Spark on a desk can and cannot be trusted to do, measured honestly. Next: the review bot I pay for read 28 of my pull requests. How many of the bugs that shipped it flagged, and what a local gate plus a frontier triage buys instead, is the next article. Follow for the count.
Appendix: the measurements behind the rules
All numbers were taken on one DGX Spark (GB10, 128 GB unified memory), vLLM 0.27.1, Qwen3.8-27B NVFP4 on port 8011 and Nemotron-3.5-Lightning-30B-A3B NVFP4 on port 8012, both systemd units. No LLM judge is involved anywhere in this article. Raw probe and A/B JSON records are bench artifacts kept outside the published page; the tables here are the complete summary. Pull requests are named by repository letter (A to F, stable across the series) and PR number.
A. The concurrency probe
scripts/bench/concurrency_probe.py: N concurrent /v1/completions requests (no chat
template, so no thinking), prompt of about 200 characters, max_tokens and
min_tokens both 512 with ignore_eos, so every stream produces exactly 512 tokens.
Per-stream rate is the completion’s own token count over its wall time; probe
aggregate is the sum over the level’s wall; engine aggregate is the delta of vLLM’s
generation_tokens_total counter over the same wall. Power is nvidia-smi --query-gpu=power.draw sampled every 3 s.
Qwen3.8-27B, 2026-09-17, with the serial eval’s one request in the background throughout (hence engine aggregate above probe aggregate):
| streams | wall s | per-stream tok/s (median) | probe aggregate | engine aggregate | power W (mean) |
|---|---|---|---|---|---|
| 1 | 46.0 | 11.14 | 11.1 | 22.3 | 34.2 |
| 2 | 46.9 | 10.93 | 21.9 | 32.8 | 35.8 |
| 4 | 47.9 | 10.69 | 42.7 | 53.5 | 36.2 |
| 8 | 50.9 | 10.06 | 80.5 | 90.6 | 36.6 |
| 16 | 59.6 | 8.59 | 137.4 | 146.0 | 37.6 |
| 32 | 88.9 | 5.76 | 184.4 | 190.2 | 39.5 |
At 32 streams the aggregate is 16.6 times single-stream. Sustained repeats at 8 streams (four rounds, then three) read 9.15 to 10.06 per stream and 73 to 80 aggregate; the 9.15 round coincided with a prefill burst on the background eval (engine aggregate 123.8). Two sustained 32-stream rounds read 5.76 and 5.78 per stream. These sustained runs are the loads the Nsight captures in B were taken inside.
Nemotron-3.5-Lightning-30B-A3B, clean curve, 2026-09-19 23:20, both engines otherwise idle:
| streams | per-stream tok/s | aggregate tok/s | wall s | power W |
|---|---|---|---|---|
| 1 | 82.4 | 82 | 6.2 | 35.6 |
| 2 | 65.0 | 130 | 7.9 | 38.1 |
| 4 | 47.0 | 188 | 10.9 | 38.9 |
| 8 | 32.8 | 262 | 15.6 | 41.5 |
| 16 | 11.3 | 181 (engine 193) | 45.3 | 65.6 |
Nemotron, 2026-09-17, co-resident with the running Qwen3.8 eval (kept for the co-residency penalty; its single-stream 29 tok/s against 82 clean is that penalty):
| streams | per-stream tok/s (median) | aggregate tok/s | power W |
|---|---|---|---|
| 1 | 28.99 | 29.0 | 35.8 |
| 2 | 24.72 | 49.4 | 36.7 |
| 4 | 18.89 | 75.6 | 37.0 |
| 8 | 12.73 | 101.9 | 39.7 |
| 16 | 9.56 | 152.9 | 41.6 |
B. GPU metrics captures (Nsight Systems)
Command, run as root (the driver reports RmProfilingAdminOnly: 1):
nsys profile --gpu-metrics-devices=0 --gpu-metrics-frequency=10000 --duration=10 \
--sample=none --cpuctxsw=none --trace=none -o <out> sleep 10
nsys export --type sqlite <out>.nsys-repSummarized from the GPU_METRICS table. The metric set is auto-selected;
--gpu-metrics-set=gb10b and gb20x are both rejected as “not supported by GPU 0”. The
set carries 19 metrics and no DRAM bandwidth counter. Each capture is 10 s at 10 kHz,
about 100,000 samples per metric (100,369 in the first); every second of every capture
reads the same to a tenth of a point, so these are steady states. Non-compute metrics
(vertex, pixel, copy engines) read zero and are omitted. All four captures were taken
on 2026-09-17 with the serial eval’s one request in the background.
| capture | Qwen3.8 streams | aggregate tok/s | GR Active % | SMs Active % | SM Issue % (mean / p50 / p95) | Tensor Active % | compute warps in flight % |
|---|---|---|---|---|---|---|---|
| 07:16 | 1 | 22 | 99.9 | 90.1 | 3.1 / 2 / 5 | 9.8 | 16.9 |
| 07:20 | 8 | 90 | 99.9 | 90.9 | 2.8 / 3 / 5 | 6.0 | 17.6 |
| 07:23 | 8 | 90 | 99.9 | 90.7 | 2.8 | 6.0 | 17.6 |
| 07:26 | 32 | 190 | 99.9 | 91.7 | 3.6 | 7.0 | 17.7 |
Root came from a scoped sudoers rule for the profiler’s profile subcommand. The
2026-09-19 security review flagged it, because a profiler that launches an arbitrary
command as root is a root shell; it was removed on 2026-09-22 and no longer exists. The
alternative, the driver module option NVreg_RestrictProfilingToAdminUsers=0, needs a
reboot and was not applied.
C. Real-cell A/B
Eval runner with --concurrency N, one angle (correctness), one seed, fixed PR sets,
scratch output directories.
Qwen3.8 thinking reviewer at 1 and 8 workers, 2026-09-19 overnight
(scripts/bench/concurrency_cells_q38_c8.sh), both engines otherwise idle (0 running
requests on either at each arm’s start). Wall: 8,507 s at 1 worker, 1,801 s at 8.
| PR | prompt tokens | 1 worker: s / output tokens / tok/s | 8 workers: s / output tokens / tok/s |
|---|---|---|---|
| B #265 | 9,857 | 510 / 5,885 / 11.54 | 1,003 / 9,233 / 9.20 |
| B #266 | 5,449 | 1,800, timeout | 573 / 5,146 / 8.99 |
| B #31 | 4,021 | 736 / 8,595 / 11.67 | 1,194 / 11,109 / 9.30 |
| B #389 | 10,953 | 958 / 11,034 / 11.52 | 1,271 / 11,918 / 9.38 |
| F #18 | 6,489 | 1,253 / 14,520 / 11.59 | 1,181 / 10,990 / 9.31 |
| C #23 | 14,117 | 751 / 8,594 / 11.44 | 430 / 3,782 / 8.80 |
| C #294 | 7,006 | 698 / 8,115 / 11.63 | 1,012 / 9,287 / 9.18 |
| E #84 | n/a | 1,800, timeout | 1,800, timeout |
Median per-cell rate 11.57 tok/s at 1 worker (n=6), 9.20 at 8 (n=7), mean 11.56 and 9.17. Total output 56,743 tokens serially and 61,465 at 8 workers. Output length is stochastic per seed, which is why the per-cell rate, not the per-cell time, is the comparable number. Both arms carry timeouts at the 1,800 s ceiling, so the wall ratio is bounded by the ceiling on both sides.
Nemotron card reviewer, 2026-09-17 (scripts/bench/concurrency_cells_ab.sh), the same
eight PRs, serial Qwen3.8 eval’s one request in the background:
| workers | wall s | cells/min | per-cell mean s | per-cell max s | per-cell tok/s median | prompt tokens (range) | output tokens (range) |
|---|---|---|---|---|---|---|---|
| 1 | 82 | 5.85 | 10.1 | 17.8 | 17.6 | 3,769 to 16,262 | 80 to 353 |
| 4 | 48 | 10.00 | 21.7 | 40.9 | 8.2 | same | 69 to 332 |
| 8 | 43 | 11.16 | 35.5 | 41.9 | 5.2 | same | 82 to 308 |
Qwen3.8 thinking reviewer at 1 and 4 workers, 2026-09-17, PRs B #267, C #294, B #265, B #454 (the first measurement, superseded by the 8-worker run above):
| workers | wall s | per-cell s (each) | per-cell tok/s (each) | output tokens (each) |
|---|---|---|---|---|
| 1 | 2,838 | 376, 821, 305, 1,335 | 11.1, 11.1, 11.0, 10.9 | 4,154; 9,128; 3,348; 14,558 |
| 4 | 1,155 | 635, 533, 897, 1,153 | 10.0, 9.9, 10.2, 10.3 | 6,350; 5,290; 9,115; 11,912 |
Long prompts, 2026-09-17 14:20: relaunching the serial eval at 4 workers reached F #88 and C #5, prompts of 40K to 44K tokens. Completed cells decoded at 7.4 to 8.4 tok/s (median of all 4-worker completions 8.08 against the 11.5 serial baseline), and 8 of 24 cells hit the 1,800 s ceiling, including every correctness cell on both PRs. The eleven timeout records from that day are quarantined and never scored. The relaunch ran light PRs (prompts up to about 17K tokens) at 4 workers and heavy ones (about 24K and up) at 2.
D. The neighbour’s own records
scripts/bench/serial_arm_tps.py over the serial Qwen3.8 arm, 2026-09-17: per cell,
usage.completion_tokens / elapsed_s, with the cell’s start reconstructed from the
record’s mtime minus its elapsed time. Timeout cells carry no tokens and are excluded
from the rate but listed.
Baseline, 113 cells overlapping no imposed load: median 11.51 tok/s, p10 10.51, p90 11.70.
| cell | window | elapsed s | output tokens | tok/s | overlapped |
|---|---|---|---|---|---|
| B #454 tests run2 | 06:39 to 06:51 | 740 | 7,056 | 9.53 | probe, 1 to 8 streams |
| B #454 tests run3 | 06:51 to 07:10 | 1,156 | 9,707 | 8.39 | probe, 16 to 32 streams |
| B #454 quality run3 | 07:19 to 07:30 | 653 | 5,878 | 9.01 | sustained 8 and 32 streams |
| B #454 security run2 | 07:36 to 08:06 | 1,800 | timeout | Nemotron eval, 4 workers | |
| B #454 security run3 | 08:06 to 08:24 | 1,083 | 5,690 | 5.25 | Nemotron eval, 4 workers |
| F #11 correctness run1 | 08:24 to 08:54 | 1,800 | timeout | Nemotron eval, 4 workers | |
| F #11 correctness run2 | 08:54 to 09:24 | 1,800 | timeout | Nemotron eval, 4 workers | |
| F #11 correctness run3 | 09:24 to 09:42 | 1,072 | 11,763 | 10.97 | Qwen3.8 A/B, 1 worker |
| F #11 tests run1 | 09:42 to 09:48 | 351 | 3,563 | 10.14 | Qwen3.8 A/B, 1 then 4 workers |
| F #11 tests run2 | 09:48 to 09:50 | 152 | 1,545 | 10.14 | Qwen3.8 A/B, 4 workers |
| F #11 tests run3 | 09:50 to 09:52 | 101 | 1,010 | 10.01 | Qwen3.8 A/B, 4 workers |
| F #11 quality run1 | 09:52 to 09:58 | 359 | 3,761 | 10.47 | Qwen3.8 A/B, 4 workers |
| F #11 quality run2 | 09:58 to 10:02 | 248 | 2,711 | 10.92 | Qwen3.8 A/B, 4 workers |
Timeouts at the 1,800 s ceiling on this arm: 5 of 72 cells on 2026-09-16 (at full speed, kept as protocol results) and 4 of 56 on 2026-09-17, three of the four inside the Nemotron window. Those three were re-run alone on a quiet engine: 1,203, 1,132 and 749 s, all real verdicts.
E. The finder’s deadline
The deep review pass runs three seeds of the same prompt concurrently against Qwen3.8, each with a 32,768-token output cap and an 1,800 s ceiling. At about 10.5 tok/s per seed with three sharing the engine, a seed can emit about 19K tokens (10.5 × 1,800) before the ceiling, so the cap is unreachable and a seed that sprays past 19K always times out. Streaming with a wall-clock deadline and salvage of complete findings from a cut-off body merged on 2026-09-22 (PR A #494).
The first job at 8 Qwen3.8 streams, the finder-context arms of 2026-09-22, shows what the ceiling does at that depth. Cells cut off at the ceiling, per arm, against the 4-stream baseline of 2026-09-14 (which ran before salvage, so its cut-offs returned nothing):
| arm | streams | real cells | complete | cut off, salvaged | nothing salvageable |
|---|---|---|---|---|---|
| baseline | 4 | 93 | 88 | n/a | 5 |
| arm 1 | 8 (14 for about 75 minutes) | 72 | 52 | 20 | 0 |
| arm 2 | 8 | 75 | 41 | 29 | 5 |
| arm 3 | 8 | 93 | 51 | 37 | 5 |
The arms also changed the prompt, and two of them produced prose inside a JSON field that no salvage can recover, so the cut-off counts are partly load and partly the prompt. They are recorded here, not used as evidence in the acts.
The ceiling experiment, 2026-09-19: the deep pass at 3,600 s instead of 1,800 s on the four PRs whose cells timed out most (5 ground-truth bugs), 2 workers on Qwen3.8, judged by Claude Sonnet 5.
| at 1,800 s | at 3,600 s | |
|---|---|---|
| bugs caught | 1 of 5 | 2 of 5 |
| cells with findings | 7 of 12 | 7 of 12 |
| cells failed | 5 timeouts | 5 hit the 32,768-token output cap (52 to 57 min each) |
| precision per run | 0.019 | 0.008 |
F. Instruments, verdicts
| instrument | on the GB10 | verdict |
|---|---|---|
nvidia-smi utilization.gpu |
96% under any decode load | resident-kernel flag, not load |
nvidia-smi power.draw |
34 to 40 W across a 17× throughput range on Qwen3.8; 66 W only when Nemotron thrashed at 16 streams | not load |
nvidia-smi dmon memory column |
reads 0 | unsupported on unified memory |
| DCGM 4.7.0 profiling | installs; the Profiling module fails to load as root and as its service account (NVPW_DCGM_LoadDriver returned 1, core log “gpuId 0 was not a GPM GPU”) |
cannot profile this part; kept as a health monitor |
| Nsight Systems GPU metrics | works as root, one metric set, no DRAM counter | the working gauge for SM activity |
vLLM /metrics counters |
generation_tokens_total, num_requests_running |
the working gauge for throughput |
| the job’s own records | tokens over elapsed per cell | the working gauge for cost to a neighbour |
G. The June curve, for comparison
From the serving article’s appendix: the retired Qwen3-Next-80B-A3B driver under llama-benchy 0.3.8 on vLLM 0.21.0, 2026-06-11, prompt 2,048 tokens, decode 512.
| concurrency | aggregate decode tok/s | per-stream tok/s |
|---|---|---|
| 1 | 37.9 | 37.9 |
| 8 | 173.4 | 22.0 |
| 16 | 243.5 | 16.4 |
Same shape, 4.6× at 8. The curve was published in June and the eval runner stayed serial until 2026-09-17.
What is not published, and why
The review outputs the eval cells produced stay private because they quote application source; this article uses only their token counts, timings, and verdict labels. The repositories are named by letter. The probe and runner scripts are the lab’s own tooling and stay private; their settings are above.
Written under Spark Bench style v0.5