NVFP4 vs FP8 vs MXFP4 vs EXL3: The Quantization Format Guide for Local LLMs

GLM-5.3-Flash ships with FP8 weights that occupy roughly 328 GB. A 4-bit build of a comparable frontier MoE, Qwen3.8-Flash in NVFP4, lands at roughly 135 GB. Same class of model, two completely different procurement decisions. One is a rack; the other is two small boxes on a desk, and a slightly smaller file would have made it one.

Quantization format is now the first question you ask about a release, ahead of context length, license and benchmark scores. Architecture decides how good a model can be; quantization decides whether you get to run it at all. This is the sizing reference: what each format is, what it costs per parameter once you count metadata, and which to pick for the hardware you own.

Bits per weight is a headline, not a file size

Every modern low-precision format stores two things: quantized elements, and the scales needed to reconstruct them. The advertised bit count describes only the first. Reconstruction is always a variant of value = element x scale, with the scale shared across a block of adjacent elements; smaller blocks track local dynamic range more closely and cost more metadata. The honest figure is:

effective_bits = element_bits + (scale_bits / block_size) + (zero_point_bits / block_size)
effective_bytes_per_param = effective_bits / 8

Run it for the formats you will actually encounter:

• NVFP4: 4-bit E2M1 elements, blocks of 16, one FP8 E4M3 scale per block. 4 + 8/16 = 4.5 bits, or 0.5625 bytes per parameter.
• MXFP4: 4-bit E2M1 elements, blocks of 32, one 8-bit E8M0 power-of-two scale per block. 4 + 8/32 = 4.25 bits, or 0.53 bytes per parameter.
• GPTQ INT4 at group size 128: 4-bit integers, one FP16 scale and one 4-bit zero point per group. 4 + 16/128 + 4/128 = 4.16 bits, or 0.52 bytes per parameter.

The formula misses a second source of inflation: not every tensor gets quantized. Embeddings and the output projection are kept at higher precision because they are disproportionately sensitive, norms and biases stay in BF16 or FP32, and MoE router weights are almost always left alone. That is why a 320B model at nominally one byte per weight comes out at 328 GB, not 320 GB. The 2.5% gap is the un-quantized remainder plus alignment padding.

So: the published file size is ground truth, everything else is an estimate. Divide repository size by parameter count for the real bytes-per-parameter. For GLM-5.3-Flash in FP8 that is 328 / 320 = 1.025. Use that ratio, not the nominal one, when extrapolating within a family.

FP8: E4M3 and E5M2

FP8 is an 8-bit float in two flavours, differing in how the bits split between exponent and mantissa. E4M3 gives 4 exponent bits and 3 mantissa bits: narrower dynamic range, finer resolution within it. E5M2 gives 5 and 2: wider range, coarser steps. E4M3 carries weights and forward-pass activations, where values are reasonably bounded; E5M2 is for training gradients, where range matters more than precision. For inference you want E4M3.

Native tensor-core support arrived with Ada Lovelace and Hopper and continues through Blackwell, so the matmul runs in FP8 end to end rather than dequantizing to FP16. FP8 is not just smaller, it is faster on the same silicon.

Quality is why it dominates. Eight bits with per-tensor or per-channel scaling leaves enough headroom that most published FP8 checkpoints are treated as interchangeable with the BF16 original. It is what labs ship when they want one artifact everyone trusts.

Sizing figure: 1.0 to 1.05 bytes per parameter.

NVFP4

NVFP4 is NVIDIA's 4-bit float: E2M1 elements (one sign bit, two exponent, one mantissa) with a per-block FP8 E4M3 scale over blocks of 16, plus a per-tensor FP32 scale. The two-level scaling is the interesting part. The block scale is itself a float rather than a power of two, so it can land anywhere in range, and the tensor-level factor keeps block scales from saturating.

Blackwell tensor cores execute NVFP4 natively: roughly half the footprint of FP8 with a real throughput gain, which is the combination that justifies the accuracy risk.

The caveat that catches people out repeatedly: a model having FP8 weights published does not mean an NVFP4 build exists. A good NVFP4 checkpoint takes calibration data and a quantization run, and labs do not always ship one. GLM-5.3-Flash has no NVFP4 release as of writing, so if you sized hardware assuming a 4-bit build would land, you are serving 328 GB of FP8. Check the hub before the purchase order.

Sizing figure: 0.56 bytes per parameter, before un-quantized tensors.

MXFP4

MXFP4 is the Open Compute Project's microscaling format, the cross-vendor answer to NVFP4. Same 4-bit E2M1 element, but blocks of 32 and a scale encoded as E8M0: eight exponent bits, no mantissa, so the scale is strictly a power of two.

Both differences push the same way. Larger blocks and a smaller scale encoding mean less metadata: 4.25 bits per weight against NVFP4's 4.5. A power-of-two scale is also cheaper in hardware, since applying it is an exponent adjustment rather than a multiply. The cost is precision. Thirty-two elements share one coarse scale, so a block with a single large outlier loses resolution on everything else, and NVFP4 generally reconstructs closer to the original tensor.

MXFP4's advantage is political rather than numerical: an open standard supported across Blackwell and AMD's CDNA 4 generation, and the one to prefer if you care about not being locked to a single vendor's tensor cores.

Sizing figure: 0.53 bytes per parameter.

INT8 and INT4: GPTQ and AWQ

The integer lineage predates the float formats and still has the most checkpoints by count. INT8 with per-channel scaling is close to free in quality terms and has had tensor-core support since Turing. INT4 is where the engineering lives. GPTQ works layer by layer, using second-order information from a calibration set to pick rounding directions that minimize error in that layer's output rather than in its weights. AWQ instead finds the small fraction of weight channels that matter most, judged by activation magnitude rather than weight magnitude, and scales them up before quantizing so they land on finer steps. Both need calibration data that resembles your workload.

The limitation: INT4 kernels mostly dequantize to FP16 and run the matmul there, so you get the memory saving and no compute saving. Fine on a bandwidth-bound single stream, less fine when saturating a large GPU with concurrent requests.

Sizing figure: 0.52 to 0.60 bytes per parameter depending on group size.

GGUF K-quants

GGUF is llama.cpp's container, and the K-quant schemes inside it (Q4_K_M, Q5_K_M, Q6_K and relatives) are why llama.cpp owns CPU and Apple Silicon inference. A K-quant is not one precision but a per-tensor policy: projections at the nominal bit width, sensitive tensors bumped a level, embeddings and output handled separately. The _M and _S suffixes are literally which mix you get.

K-quants also use two-level scaling inside each super-block, quantizing the block scales themselves, which buys back accuracy at a given bit rate. The cost is that the bit rate is emergent rather than declared, so a Q4_K_M file is meaningfully larger than 4 bits per weight suggests. There is no tensor-core path either: K-quants dequantize into SIMD registers and run the matmul on AVX2, AVX-512, NEON or Metal. With unified memory and no discrete GPU, that is the right trade.

Sizing figure: use the actual GGUF file size. Do not estimate this one.

EXL3

EXL3 is the format of ExLlamaV3, and its target is specific: consumer NVIDIA cards, single stream, maximum tokens per second for one user. Trellis coding rather than simple block scaling lets it hit fractional bit rates and reconstruct better than naive round-to-nearest INT4 at the same size.

The community moves faster than vendors here. A GLM-5.3-Flash-EXL3-Q4 quantization exists, which is how people run that model on hardware that cannot hold 328 GB. Independent testing put a tuned GLM-5.3-Flash setup at roughly 800 tokens per second, the kind of number that makes the quality trade worth arguing about.

EXL3 is not for multi-tenant serving: batching is weaker than vLLM or TensorRT-LLM, and it is CUDA-only. For one person on two or four consumer cards, it is frequently the fastest thing available.

Sizing figure: bits per weight is configurable; at 4 bpw, roughly 0.53 bytes per parameter.

The comparison table

The sizing arithmetic, worked

Weights first:

weights_bytes = parameters x bytes_per_param

Then add everything that is not weights. KV cache scales with batch size, context length, layer count and KV head count, and is a separate budget; activation buffers, CUDA graphs and allocator slack take another cut. Budget 10 to 30 percent above the weight figure before calling a model a fit, the high end for long contexts and many concurrent requests.

Example 1: GLM-5.3-Flash in FP8. 320B parameters at the observed 1.025 bytes each gives 328 GB of weights. Add 20 percent headroom: roughly 394 GB of usable memory. Four 128 GB DGX Spark units give 512 GB, so it fits with room for context, and at TP=4 they deliver roughly 1092 GB/s aggregate. A 512 GB M5 Ultra holds the same weights in one box at roughly 1200 GB/s: comparable capacity, slightly more bandwidth, no interconnect between shards.

Example 2: Qwen3.8-Flash in NVFP4 against a 128 GB box. The NVFP4 weights are roughly 135 GB, already 7 GB over a single 128 GB DGX Spark before you allocate one KV cache page. The answer is two units, not one. A 4-bit release is not automatically a single-workstation model: check the number, not the adjective.

Example 3: a build that does not exist yet. GLM-5.3-Flash at 4 bits per weight, using 0.53 bytes per parameter, projects to roughly 170 GB plus whatever the quantizer left in higher precision. That is the arithmetic that makes a community EXL3 Q4 build interesting on hardware that cannot hold 328 GB. Treat it as a planning estimate, replaced by the real file size the moment a build lands.

Weight-only versus activation quantization

Most of the above is weight-only quantization: weights live in low precision, get dequantized into the compute type, and the matmul runs in FP16 or BF16. That saves memory and, on a bandwidth-bound workload, time, since the bottleneck is streaming weights rather than the multiply.

Quantizing activations too unlocks the native low-precision tensor-core path. Both operands are FP8 or FP4, the matmul itself is cheaper, and that matters once you are compute-bound rather than bandwidth-bound: large batches, long prefills. It is also harder. Activations have outliers that weights do not, they change with every input, and scales must be calibrated ahead or computed on the fly. A checkpoint labelled "FP8" may be FP8 weights with BF16 activations, or FP8 for both. Read the config before predicting throughput.

Why MoE breaks everyone's mental model

This is the single most misunderstood point in local LLM sizing, so state it plainly.

GLM-5.3-Flash is 320B total parameters with 18B active. The 18B figure describes compute: for any token the router selects a small subset of experts and only those multiply. It tells you how fast the model runs.

It tells you nothing about memory. Every expert must be resident, because any token may route to any of them, and you cannot page experts in on demand at interactive latency. The memory requirement follows the full 320B, which is why FP8 weights come to 328 GB rather than the roughly 18 GB an 18B dense model would need.

For MoE models: size memory on total parameters, size throughput on active parameters. Sparsity buys small-model speed at dense-model memory cost. Good trade, provided you budgeted the memory.

Picking a format for your hardware

Blackwell (RTX PRO 6000, B200, DGX Spark). NVFP4 when a build exists; you paid for the tensor cores that execute it. FP8 when it does not, which is more often than the marketing suggests. MXFP4 if the workload may move onto AMD later.

Ada and Ampere (4090, A100, L40S). FP8 on Ada, since the tensor cores are there. Ampere has no FP8 path: INT8 for near-lossless serving, INT4 via GPTQ or AWQ when the model will not otherwise fit. For one user rather than a service, EXL3 beats both.

Apple Silicon. GGUF K-quants through llama.cpp or MLX. Q4_K_M when memory is tight, Q6_K when you have headroom, because the difference on code and tool calling is worth the gigabytes. Unified memory makes capacity generous and bandwidth the constraint, the regime where a smaller quant wins on speed as well as fit.

CPU only. GGUF, and be realistic about bit rate. Q4_K_M is the floor for anything you will trust. Memory bandwidth is the bottleneck, so fewer bits is directly faster, but the quality cliff below 4 bits arrives sooner than you want.

What degrades first

Quantization does not degrade a model uniformly. General conversational quality is robust, which is the trap: the model still sounds fine while the capabilities you deployed it for are already broken. The usual order of failure:

1. Long-context coherence. Retrieval from the middle of a long context goes first: the model paraphrases rather than quotes, and drops constraints stated early in a document.
2. Instruction following. Multi-part instructions with negative conditions ("do not include X") get partially dropped, and output format drifts over turns.
3. Code generation. Syntax survives, correctness does not: off-by-one errors, wrong API signatures, plausible calls to functions that do not exist.
4. Tool calling and structured output. The most brittle, and the one that breaks agent pipelines: malformed JSON, hallucinated parameter names, arguments of the right type and the wrong value. If you run an agent loop, this is your canary.

A perplexity delta surfaces none of this. Perplexity is dominated by easy tokens, which a quantized model predicts about as well as the original. Evaluate on your own workload. Take 100 real requests from your logs, run them through both builds, and compare on the axis you care about: schema validity rate, test pass rate, retrieval accuracy. An afternoon of work, and the only evidence worth acting on.

Checklist

• Get the real file size from the repository and divide by parameter count. That is your bytes-per-parameter, not the nominal one.
• For MoE models, size memory on total parameters and throughput on active parameters.
• Add 10 to 30 percent over the weight figure for KV cache, activations and allocator slack before declaring a fit.
• Confirm the quantization you are planning around exists as a published build. FP8 does not imply NVFP4.
• Match format to silicon: NVFP4 or MXFP4 on Blackwell, FP8 on Hopper and Ada, INT4 on Ampere, EXL3 for single-stream consumer NVIDIA, GGUF on Apple Silicon and CPU.
• Check whether "FP8" in the config means weights only or weights and activations. It changes the throughput story.
• Evaluate on your own traffic, weighted toward tool calls, structured output and long-context retrieval. Ignore perplexity deltas.
• Keep the higher-precision build available. When something subtle breaks in production, swapping back is the fastest way to learn whether quantization caused it.