A stability-first SGLang deployment serves DeepSeek V4.1 Flash across three NVIDIA DGX Sparks: 256k context limit, 750k-token KV pool, ~38 tok/s single-stream decode.
Correction (2026-10-11): An earlier version of this article said the default context was 500k tokens and that prefill ran at roughly 1600 to 1700 tok/s at every context length. Neither matched the source. The configured limit is 256k tokens per request, with about 32k usable on the head node; the 750k-token KV pool was correct. The repo's README gave no prefill rate at publication; it now lists about 2,000 tok/s for three Sparks (TP=3). The 1600 to 1700 figure has been removed.
DeepSeek V4.1 Flash now runs on a cluster of three NVIDIA DGX Spark units behind an OpenAI-compatible endpoint. The deployment repo ships conservative defaults chosen for stability rather than peak numbers, and the throughput figures below come from that default profile. The serving stack itself, SGLang, is one of three engines compared in choosing a production inference engine: SGLang vs vLLM vs TensorRT-LLM.
Default settings
The defaults are deliberately conservative so the cluster stays stable under sustained load:
- 256k per-request context limit (about 32k usable in practice, memory-bound on the head node), 750k-token KV pool
- Prefill: the repo's README gave no rate when this article was published; its current version lists about 2,000 tok/s for three Sparks (TP=3)
- Around 38 tok/s of prose decode in a single stream
- Around 79 tok/s of aggregate prose decode across 4 concurrent streams
How the cluster is built
The serving stack is SGLang with FlashInfer kernels. Expert weights are stored in MXFP4 and the dense weights in FP8, with DSpark speculative decoding enabled. The tradeoffs between those two quantization formats are covered in NVFP4 vs FP8 vs MXFP4 quantization VRAM math. The model runs tensor-parallel across three GB10 nodes, each with 121.7 GiB of unified memory, joined in a ConnectX-7 RoCE triangle. The Engram n-gram tables, about 189 GiB, are offloaded to local NVMe on every node, and the Engram cache is disabled by default because reuse measured close to zero. A separate profile supports a four-Spark TP4 layout.
Decode throughput by stream count
Aggregate decode climbs as streams are added, while per-stream speed falls. Measurements are with TP=3 and DSpark speculation on: The mechanism behind that speedup is explained in speculative decoding and multi-token prediction for faster LLM generation.
Aggregate decode throughput by concurrent stream count, TP=3 with DSpark speculation.
| Streams | Aggregate tok/s | Per stream |
|---|---|---|
| 1 | 37.9 | 37.9 (TTFT 248 ms) |
| 2 | 58.9 | 30.5 |
| 3 | 71.2 | 24.5 |
| 4 | 78.6 | 20.9 |
The README states that 32k prompts (4 concurrent) are verified and that 64k and above exhausts the head node. Separately, long-context behaviour was verified with a single 208k-token prompt and with four concurrent 46k-token prompts. The VRAM arithmetic behind that KV cache growth is broken down in why context length costs more VRAM than model weights.
Getting started
Bring the cluster up with the bundled launcher, in order:
./start.sh doctor # validate nodes and links
./start.sh build # build the image on every node
./start.sh share # export the checkpoint over NFS
./start.sh pack # cache Engram shards to local NVMe
./start.sh serve # start workers, then the head node
A full boot takes 12 to 13 minutes.
Get it here: MiaAI-Lab/DeepSeek-v4.1-Flash-DGX-Sparks