watch Local Llm Inference · Research

Edge0 Claims Its 145B MoE Runs on One 24 GB GPU at 79.6 Tokens a Second

Data graphic: Edge0-145B, a 145B model on one card. A huge 24 GB stands for the VRAM of one RTX 5090 D; beside it, 38.27 GiB of host RAM also needed for NVFP4, 79.6 tok/s chat decode and 10.12B parameters active per token. Vendor-reported figures.
OP

AI security researcher · Published Nov 16, 2026, 11:24 AM EST

Edge0-145B parks its experts in host RAM to fit a 24 GB RTX 5090 D. The catch: 31 to 38 GiB of system RAM and a custom licence. All figures are vendor-reported.

Edge0 has put a 145-billion-parameter mixture-of-experts (MoE) model, Edge0-145B, on Hugging Face and says it runs on a single 24 GB consumer card, an RTX 5090 D, at 79.6 tokens a second in chat. The trick is expert offload: the GPU holds what fits and the experts sit in host RAM. Every figure in this article is vendor-reported, taken from Edge0's own model cards, and none has been independently reproduced. If you want to try it, the number to check first is not your VRAM but your free system RAM: the NVFP4 build used 38.27 GiB of it on top of the card.

What Edge0-145B is

The model has 145B parameters in total and activates 10.12B per token. Edge0's card lists 48 layers, 512 experts per layer (10 routed plus 1 shared per token) and a 256K context window.

It is built on Qwen/Qwen3.8-Flash-Next. Edge0 says it is not a re-quantisation of that model: the routed experts were replaced with a latent MoE and the result was post-trained. It ships in two builds, NVFP4 (47.79 GiB of weights) and a Q3_K GGUF (40.81 GiB). If the format names are new to you, our NVFP4 vs FP8 vs MXFP4 guide walks through the bytes-per-parameter math.

The memory you actually need

With expert residency on, Edge0 measured this on the RTX 5090 D 24 GB:

BuildWeightsVRAM usedHost RAM used
NVFP447.79 GiB23.25 GiB38.27 GiB
Q3_K40.81 GiB23.14 GiB31.28 GiB

So a 24 GB card is necessary but not sufficient. Plan for the card to be nearly full and for 31 to 38 GiB of system RAM to be taken by the model, plus whatever your OS and other apps need.

Edge0 also ran a tighter setup: 16 GiB of VRAM and a 20 GiB RAM cap (enforced with a cgroup MemoryMax=20G, no swap). On Q3_K it reports 36.8 tok/s chat decode there, against 18.5 for Qwen3.8-Flash-Next under the same budget. The card calls 16 GiB VRAM plus 20 GiB RAM test limits, not minimums, so do not read them as a supported floor. For machines short on RAM, a --resident-budget-gib 20 option (it requires --expert-profile) reads uncached experts from SSD instead.

Speed by machine

Edge0 reports two decode figures per machine: chat and agent. The headline 79.6 is chat decode on NVFP4; agent decode on the same card is 98.3. The faster number on the 5090 D comes from the smaller Q3_K build, not NVFP4.

Data graphic: lollipop chart of Edge0-145B chat decode by machine and build: RTX 5090 D Q3_K 99.7 tok/s, RTX 5090 D NVFP4 79.6 the headline figure , Apple M2 Ultra Q3_K 60.9, RTX Spark N1X NVFP4 40.1.

Chat decode in tokens per second by machine and build, as reported on Edge0's model cards. The headline 79.6 is the NVFP4 build; the Q3_K build is faster on the same RTX 5090 D.

MachineBuildChat decode (tok/s)Agent decode (tok/s)
RTX 5090 D 24 GBNVFP479.698.3
RTX 5090 D 24 GBQ3_K99.7121.2
RTX 4090 24 GBQ3_K69.991.7
RTX Spark N1X 60 GiBNVFP440.146.0
RTX Spark N1X 60 GiBQ3_K37.736.7
Apple M2 Ultra (Metal)Q3_K60.957.0

Prefill on NVFP4 on the 5090 D is 1839.6 tok/s at 4K and 3165.7 at 16K. The M2 Ultra manages 755.6 and 800.3 on Q3_K. Apple Silicon gets the Q3_K build only: the card says NVFP4 is CUDA-only.

For how this compares with models that fit entirely in memory, see our picks for 16 GB cards and the Mac mini M4 and for one to four DGX Sparks.

Benchmarks: a claim to check

Edge0 reports 86.5 on Terminal-Bench 2.1 against 83.3 for Qwen3.8-Flash-Next, and 64.6 on DeepSWE 1.1 against 58.7. Both are self-reported on the model card, with no independent evaluation or methodology given. Treat them as a claim, not a ranking.

The licence catch

Edge0-145B is under the Edge0 Community Model License 1.0, a custom licence, not open source in the OSI sense. The Qwen materials stay under the Qwen Community License 1.0.

It allows personal use, research, modification, noncommercial sharing, internal use and commercial use of the model's outputs. Selling or bundling the model, or running it as a commercial model service, needs a separate written licence from Edge0. The framework code is Apache-2.0 and Strata is MIT.

What to do before you download

  • Check free system RAM first: about 31 GiB for Q3_K or 38 GiB for NVFP4, on top of a 24 GB card.
  • Use Edge0's own engine (edge0-infer with edge0-strata), on CUDA 13 under Linux or Windows, or Metal on macOS for Q3_K. It serves OpenAI- and Anthropic-compatible APIs on 127.0.0.1:8000.
  • Ignore the Hugging Face sidebar metadata: it shows 5B params and :BF16 snippets that do not match the 145B model.
  • Read the licence before you offer the model as a hosted service or bundle it in a product.
  • Run your own agent tasks before trusting the Terminal-Bench figure.

Sources

Keep reading

All latest →
  1. watchResearchQwen4-Max Preview Is Token Plan Only and Retires 25 Nov: Build on qwen4-max Instead4 min
  2. watchResearchGitHub Copilot Drops Six Models on Dec 14: What Replaces Fable 5, Haiku 4.5 and Gemini 4 Argon5 min
  3. watchResearchClaude Max 20x Is Now Ultra: No Five-Hour Limit, if Your Account Is Verified4 min
  4. watchResearchCopilot Fusion Goes GA: Per-Model Billing, 30% Off, and How It Compares With Claude Code Max4 min
  5. stableResearchRipple-65B Preview: A Dense 65B Open Model That Fits One DGX Spark and Claims 80.5 on Terminal-Bench 2.17 min
  6. elevatedResearchGemini 2.5 Pro and 2.5 Flash API Calls Reroute on 1 December: What Changes in Your Bill and Your Code4 min