Best Local AI Models to Run on Your DGX Sparks

NVIDIA's DGX Spark ships with 128 GB of unified memory on a GB10 chip, two ConnectX-7 ports, and a price that puts it on a desk rather than in a rack. That memory ceiling is the whole game: a single box holds one frontier-class open-weight model if the checkpoint is quantized hard enough, and every extra Spark you cable in roughly doubles the weights you can keep resident. Below is the model we would pick at each node count, based on the recipes and published measurements from Mia's AI Lab, which has spent the last month wringing usable tokens per second out of one-, two-, three- and four-Spark fleets.

All throughput figures are the lab's own sparkDash measurements, quoted from each repo's README as of 2026-09-12. They are single-box or small-cluster numbers with speculative decoding on; treat them as what a competent operator gets on day one, not as a hardware ceiling.

The short version

One Spark: Qwen3.8-Flash-Next

Recipe: MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark

A lone Spark has about 121 GiB of usable unified memory, and the lab's NVFP4 build of Qwen3.8-Flash-Next is a 99 GiB checkpoint. That fits only because the recipe offloads the model's PLE table (a further ~27 GiB packed on first launch) and memory-maps it instead of holding it on the GPU budget. Budget roughly 130 GiB of free disk and 10 to 12 minutes from ./start.sh to a healthy endpoint on port 8888.

What you get is a vision-language model that takes text, images and video out of the box, with a 262,144-token context by default (YaRN unlocks 512k at the cost of a smaller KV pool). The shipped profile runs MTP 3 speculative decoding with FP8 KV cache and leaves about 16.5 GiB of KV, which is roughly 992k tokens or 3.8 concurrent full-length requests.

Measured decode, prose prompts, 2026-09-06:

Prefill lands between 1,500 and 1,900 tok/s depending on chunk size and KV dtype, which means a 400k-token prompt takes about four and a half minutes to first token. The README is candid about the one soft spot: decode under a concurrent long prefill drops to single-digit tok/s for the duration of that prefill, because vLLM emits one chunk per engine step and the step time balloons. If your workload mixes short chats with 100k-token document dumps, expect that.

The recipe also documents two lost servers from an earlier, more aggressive KV target, which is why the current default caps the host side at 26 GiB reserve. Read that section before raising KV_TARGET_GIB.

Two Sparks: three good answers

Two Sparks over a direct ConnectX RoCE link is the sweet spot of the range. Weights split across two ranks with tensor parallelism, the head node exports its Hugging Face cache over NFSv4 so the worker never needs its own copy, and all three recipes below serve one OpenAI-compatible endpoint from the head.

Two DGX Sparks over ConnectX RoCE: three serving recipes and their measured single-stream decode.

DeepSeek V4 Flash (Vision-Exp)

Recipe: MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark

This is the pick if you want the deepest context on two boxes. The recipe runs the official deepseek-ai/DeepSeek-V4-Flash-Vision-Exp checkpoint under vLLM TP=2 with DSpark speculative decoding (MTP_NUM_TOKENS=6), a 1,048,576-token model ceiling, and DeepSeek's nvfp4_ds_mla packed KV cache. On the lab's pair that KV pool is 17.04 GiB, or 2,331,430 tokens, enough to hold two full 1M-token conversations.

Throughput: a single chat decodes at roughly 62 to 83 tok/s after first token at any prompt length up to 128k; six short chats at once run at about 160 to 190 tok/s aggregate. The trade-off is prefill scheduling. Long prompts are chunked and only one prefill runs at a time by default, so six cold 128k prompts arriving together mean minutes of time-to-first-token for the last one in the queue, with decode for existing streams dropping to an ~8 tok/s floor while it churns. There is an opt-in switch for two overlapping prefills.

The same repo ships a separate three-Spark launcher (TP=3, 16 slots) that lifts aggregate decode to about 200 tok/s at 16 streams but makes prefill 4 to 22 percent slower, so treat it as a concurrency option rather than a latency one.

GLM-5.3 Flash (EXL3 4 bpw)

Recipe: MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks

The lab's own EXL3/TR3 quant of GLM-5.3 Flash at a uniform 4 bits per weight is about 164 GiB across 120 shards, served with fp8 KV and a DFlash2 k=7 draft model for speculation. This is the recipe with the most engineering in it: a custom grouped-MoE CUDA path (the E3 kernel) that lifted cold prefill roughly 40 percent to about 1,580 to 1,640 tok/s all the way out to 256k-token prompts, plus two opt-in decode accelerators.

Decode numbers depend heavily on what you generate. On structured and code output, where the drafter's acceptance rate is high, a single stream runs at 62.9 tok/s and four streams reach 146.5 tok/s aggregate. On prose the stock configuration measured 18 to 27 tok/s per stream; turning on adaptive verification length plus FP8 weight-only dense projections lifts that to 32.1 tok/s single-stream. The FP8 dense path is marked provisional because it changes target numerics slightly (argmax agreement 94 to 100 percent against the BF16 reference), so leave it off if you need a bit-exact serve.

Default context shipped at 850k tokens with a memory utilization of 0.85. The README notes that prompts of 100k tokens and above run close to the head node's host-memory limit at any setting, which on a Spark means GPU memory, because they are the same pool.

Qwen3.8-Flash-Next (dual)

Recipe: MiaAI-Lab/Qwen3.8-Flash-Next-Dual-DGX-Sparks

The same NVFP4 Qwen checkpoint as the single-box recipe, now split TP=2 with expert parallelism and MTP 3. The extra box mostly buys KV: with weights halved per rank, vLLM allocates 32 GiB of KV cache, a 3,652,200-token pool, and the model runs at its full 1M context. Decode improves modestly on a single stream and substantially under load:

Prefill is the standout: flat at about 2,960 tok/s from 16k to 64k prompts and still 2,727 tok/s at 128k, which is nearly double the single-box figure. If your use is agentic coding with large repository context, this is the fastest time-to-first-token in the two-Spark set. The recipe also carries an optional path for Qwen's official FP8 checkpoint, at the cost of dropping back to native 262k context and a ~500k-token KV pool.

Three Sparks: DeepSeek V4.1 Flash (TP=3)

Recipe: MiaAI-Lab/DeepSeek-v4.1-Flash-DGX-Sparks

DeepSeek V4.1 Flash is a 476 GiB checkpoint on disk, and this is the first model on the list that genuinely does not fit on two Sparks: two ranks would need 145 GiB of resident weights each. The lab's answer is to move the model's two Engram n-gram tables (189 GiB) onto each node's local NVMe and serve them through a host callback, leaving about 305 GiB of MXFP4 experts and FP8 dense weights to split three ways at roughly 101 GiB per rank. Three Sparks form a full ConnectX-7 triangle, so no switch is needed. The stack is SGLang rather than vLLM, with DSpark speculative decoding, tool calling and vision behind one endpoint.

The awkward part is that 64 attention heads, 8 output groups, the 129,280-token vocabulary and the 128 DSpark draft experts are not divisible by three. The adapter pads heads to 96 and groups to 12, which means rank 2's attention shard is entirely zero-filled padding. It works, and greedy decoding is deterministic run to run, but you are paying for GEMMs on empty tensors.

Measured on the three-node fleet:

Context is configured at 200k to 256k against the model's 1M maximum, with a 750k-token KV pool and only about 6 GB of memory left on the head while serving. The README is blunt that anything pinning host memory on the head, NCCL buffers included, stalls all three ranks because TP is synchronous. A full boot is 12 to 13 minutes, eight of them spent reading the checkpoint.

Four Sparks: DeepSeek V4.1 Flash (TP=4)

Same repo, separate start-tp4.sh launcher and .env.tp4 profile. At TP=4 every awkward dimension divides cleanly, so there are no padded shards, resident weights fall to about 77 GiB per rank, and about 40 GB per rank is free while serving. That headroom is what the profile spends: the full 1,048,576-token model context, a 4,000,000-token KV pool, and CUDA graphs for eight concurrent requests instead of four.

Two caveats. First, a Spark has only two ConnectX-7 ports, so four nodes cannot form a full mesh; you need a RoCE switch or a ring with NCCL routed over it, and the doctor script checks reachability before committing to a 13-minute boot. Second, and more important for anyone reading the table above: the lab has validated the TP4 profile for configuration and script paths only. It has not been booted. Expect the same per-step structure as TP3 with smaller attention GEMMs and one extra network hop per collective, which means the win is memory and concurrency, not a faster single stream.

How to choose

• Budget for one box and you want vision plus a real context window: the single-Spark Qwen3.8-Flash-Next recipe is the only mature option, and 48.7 tok/s single-stream is comfortable for interactive use.
• Two boxes, long-document or agent workloads: DeepSeek V4 Flash for the 1M context and the 2.3M-token KV pool, or dual Qwen3.8 if prefill speed matters more than raw context.
• Two boxes, code generation: GLM-5.3 Flash EXL3, whose 62.9 tok/s on structured output is the fastest single-stream figure in the set, provided you accept 4-bit weights.
• Three or four boxes: DeepSeek V4.1 Flash is the only model on the list that needs them, and at three nodes the trade is a bigger model for a slower stream. Wait for a booted TP4 benchmark before buying the fourth Spark on the strength of the 1M-context line.

Every one of these recipes runs a container from a pinned image, exports one OpenAI-compatible endpoint, and documents its failure modes with unusual honesty. Read the memory sections before changing a single knob: on a DGX Spark, running the host out of RAM and running the GPU out of memory are the same event.