SGLang vs vLLM vs TensorRT-LLM: Choosing a Production Inference Engine

Between a directory of safetensors files and an HTTP endpoint that answers a completion request sits a piece of software most teams pick in an afternoon and then live with for two years. It decides how many concurrent users a box can serve, how much accelerator memory goes to the KV cache rather than sitting reserved and idle, which quantization formats you can load, what happens when the queue grows faster than the scheduler drains it, and how long it takes to put a new model into production. For most deployments it moves cost per token further than any model or prompt change you are weighing.

Three projects dominate the GPU serving tier: vLLM, SGLang, and TensorRT-LLM. All three are good, and all three do continuous batching, paged KV cache, tensor parallelism, FP8 weights, and an OpenAI-shaped API. Feature comparisons between them go stale fast in both directions, so treat every specific capability claim here as something to confirm against current release notes before planning around it. What ages well is architecture: how execution is compiled, how the KV cache is organised, and how tightly the project is bound to one vendor's silicon. Those are the honest differentiators, and this guide is built around them.

What an inference engine actually does

Continuous (in-flight) batching. Naive batching waits for every sequence to finish before starting the next batch, wasting most of the GPU because sequences finish at wildly different lengths. Continuous batching admits new requests into the running batch at each decoding step and evicts finished ones immediately. Under real traffic it is worth several times the throughput of static batching.

KV cache management and paging. Keys and values for every token of every active sequence live in accelerator memory. One contiguous block per sequence sized to maximum context loses enormous memory to fragmentation; paging into fixed-size blocks with an indirection table removes that waste and makes sharing between sequences possible. The KV cache, not the weights, limits concurrency at long context, so how an engine manages it is the most consequential design decision it makes.

Scheduling and admission control. Which prefills run now, which decodes get a step, when to preempt a sequence whose blocks were evicted, how to stop long requests starving behind short ones. Chunked prefill, prefill and decode disaggregation, and priority policies live here. Tail latency is won or lost in this layer.

Quantization support. Weight-only INT4 and INT8, FP8 weights and activations, NVFP4 and other 4-bit float formats, KV cache quantization. Support is uneven and moves fast. The question is never "does it support quantization" but "can it load the checkpoint format I have, on the hardware I own, with the kernel that makes it fast rather than the fallback that makes it slow".

Kernel selection. Attention kernels, fused MLP and MoE kernels, gather and scatter for expert routing, tiling for your shapes and hardware generation. Much of the gap between engines on one model is which kernel path your sequence lengths select.

Distributed execution. Tensor parallelism inside a layer, pipeline parallelism across layers, expert parallelism for MoE, data-parallel attention at scale. Single-node multi-GPU is handled well everywhere; multi-node is where maturity diverges.

API surface. OpenAI-compatible chat and completions, streaming, tool calling, constrained output, metrics endpoints. Compatibility is near universal; the edges (logprobs, tool-call streaming semantics, schema coverage) are where differences show.

vLLM: the sensible default

vLLM came out of the PagedAttention work, which applied operating-system virtual memory ideas to the KV cache: fixed-size blocks, a block table per sequence, physical blocks allocated on demand and shared copy-on-write. That removed the fragmentation capping batch sizes and lifted concurrency on unchanged hardware. The idea was good enough that everyone adopted a version of it, which is why paged KV is now table stakes rather than a vLLM advantage.

What distinguishes vLLM today is reach. It supports the widest range of model architectures, usually picks up new ones fastest, and runs on more than NVIDIA, with AMD, Intel and other backends at varying maturity. The community is the largest of the three, which matters more than it sounds: when your model throws an odd error at 3am, the odds someone has already answered that issue are higher. The distance from install to serving requests is short and the defaults are sane.

It is weaker at the top end. On a fully tuned, single-model, NVIDIA-only deployment it will usually not beat a well-built TensorRT-LLM engine, and on prefix-heavy traffic it will usually not beat SGLang. Breadth has also cost it simplicity: a large configuration surface with interacting knobs, and backends outside the NVIDIA main line that see less testing.

Treat vLLM as the default you have to argue your way out of. Absent a specific reason, pick it, then revisit with production measurements in hand.

SGLang: prefix sharing as an architecture

SGLang's distinguishing idea is RadixAttention. Rather than a flat block pool with opportunistic reuse, it organises cached prefixes in a radix tree keyed by token sequence. A new request walks the tree, finds the longest already-cached prefix of its prompt, and reuses that KV state directly; eviction is LRU over tree nodes, so hot prefixes survive. The scheduler is prefix-aware too, and can order requests so those sharing a prefix run together.

The consequence is that a prompt costs roughly what its unshared suffix costs. An agent loop resends the same multi-thousand-token system prompt, tool schema block, and accumulated scratchpad at every step, appending a little at the end. A multi-turn chat resends the whole conversation with one new message. A few-shot classifier resends twenty fixed exemplars. RAG resends a fixed preamble around varying retrieved context. Most of every request is textually identical to a previous one, and a radix tree turns that from repeated prefill compute into a pointer walk. On agent traffic, good versus poor prefix sharing is not a few percent; it changes the shape of your prefill bill.

SGLang also has a strong constrained-decoding story, with grammar and JSON-schema constraints integrated into the decode loop rather than bolted on, and deliberate work on keeping that fast when many concurrent requests each carry their own schema. If your product is a JSON API dressed as a model, that matters.

It is also where much of the frontier long-context and MoE work is happening. Community testing reports a two-node DGX Spark deployment of Qwen3.8-Flash-Next reaching 900k context with vision enabled under SGLang, using multi-token prediction and an SM121 kernel patch, stress tested at a 300k token prefill, measuring roughly 64 tokens per second single stream and roughly 115 tokens per second at two to four concurrent sessions. That model in NVFP4 is approximately 135 GB, hence two 128 GB Sparks. That is community measurement, not a vendor specification. Note its shape: single-stream decode is modest, aggregate throughput climbs with concurrency, and reaching it took a kernel patch and a speculative decoding technique, which is a fair picture of frontier long-context serving generally. Community testing elsewhere reports around 800 tokens per second on GLM-5.3-Flash in a tuned serving setup, a 320B total and 18B active MoE whose FP8 weights come to approximately 328 GB, the kind of sparse model where expert parallelism and cache behaviour dominate everything else.

The tradeoff is that SGLang moves fast and its operational surface is less settled. Configuration names and defaults shift, model-specific tuning guidance often lives in an issue thread rather than stable documentation, and you should pin versions before upgrading.

TensorRT-LLM: compile ahead, run fast

TensorRT-LLM takes the other path. Instead of interpreting a graph at run time it compiles the model ahead of time into an engine artifact specialised to one model, precision, parallelism layout, hardware generation, and declared range of batch sizes and sequence lengths. That build fuses operations, autotunes kernels against the actual target device, bakes in attention and MoE plugin implementations, and leaves far less to decide at run time.

The payoff is the highest achievable performance on NVIDIA hardware, especially at high concurrency on a known shape distribution and on the newest silicon, where NVIDIA's own kernels land first. Across thousands of GPUs, a few percent is a budget line.

The price is flexibility. The build takes real time and memory, belongs in CI with artifact storage and versioning, and must be redone when the model, precision, parallelism layout, GPU generation, or shape envelope changes. Serving outside the compiled shape range is impossible or slow, a new model means a new build, and a new architecture may mean waiting for support. It is NVIDIA-only by construction, with no portability story by design.

TensorRT-LLM is right when model, hardware, and traffic pattern are fixed and the scale justifies the engineering. It is wrong while any of those are still moving.

Honourable mentions

llama.cpp and GGUF. A C++ implementation with excellent CPU support, strong Apple Silicon support via Metal, partial GPU offload, and a quantization ecosystem that fits large models into small memory with graceful quality degradation. The standard way to run a model on a laptop, a Mac Studio, or an edge box.

Ollama. A developer-experience layer over llama.cpp: model registry, one-command pulls, sane defaults, an API server, lifecycle management. It removes nearly all local setup friction.

Both target a different problem: one user, easy setup, hardware you already own. The three main engines target many concurrent users and aggregate throughput on hardware bought for the purpose. Benchmarking one against the other is a category error, and deploying Ollama as a multi-tenant production backend is an expensive version of the same error.

ExLlama. A narrower niche worth knowing: high single-stream throughput from quantized models on consumer NVIDIA cards. If your constraint is one or two gaming GPUs and one user's token rate, it is competitive in a way the big three are not tuned for.

Comparison

The differences that will outlive this release cycle

Runtime execution versus ahead-of-time compilation. vLLM and SGLang decide at run time, using JIT compilation and graph capture to claw back overhead; TensorRT-LLM decides at build time. Everything else follows. Runtime buys flexibility, fast model onboarding, and no artifact pipeline. Compilation buys peak performance and costs a rebuild whenever a variable changes. Feature parity keeps converging; this split will not close, because it is a real tradeoff rather than a missing feature.

Paged KV versus radix-tree prefix sharing. Not opposites: SGLang pages memory too, and vLLM has prefix caching. The difference is what the design is organised around. When sharing is a first-class structure the scheduler reasons about, the engine can group and order work to exploit it; when it is a lookup layer over a block pool, it helps but shapes less.

Portable versus vendor-coupled. A single-vendor engine uses that vendor's newest instructions on day one and pays for no abstraction layer. A portable one cannot, but lets you buy whatever accelerator is available and affordable next year. Given recent supply and pricing, that is a procurement decision as much as a technical one.

Choosing

One model, fixed NVIDIA fleet, throughput is the business. TensorRT-LLM. Exactly the conditions its tradeoff was designed for: the build cost amortises across a large stable deployment and the percentage points are real money. Budget the CI work up front.

Many models, or frequent model swaps. vLLM. Onboarding a new architecture is your most frequent operation, and a compile step turns every swap into a project. SGLang is a fair alternative if traffic is prefix-heavy. Do not pick the compiled option here.

Agent workloads with long shared system prompts. SGLang. This is what RadixAttention was built for, and the saving compounds with every tool-call round trip. Measure on your own traffic, but expect it to win.

Very long context and MoE. SGLang first, vLLM as the check. Both are active; SGLang carries more of the frontier work, and this is where kernel details and speculative techniques like multi-token prediction matter most. Plan on version pinning and hand-tuning either way.

Heavy structured or JSON output. SGLang, with vLLM close behind. Test your real schemas at your real concurrency: constrained decoding cost varies enormously with schema complexity and with how many distinct grammars are live at once.

Non-NVIDIA accelerators. vLLM, and verify your chip, model, and quantization combination before committing hardware budget. A supported backend is not necessarily a fast one.

A laptop or single workstation. Ollama to be working in five minutes, llama.cpp for control over quantization, offload split, and context, ExLlama on one or two consumer NVIDIA cards where single-stream speed is the point. Do not stand up a production engine to talk to yourself.

Benchmarking them fairly

Most published engine comparisons are unusable. The rules are simple and widely ignored.

Hold quantization identical: same checkpoint, same weight format, same KV cache precision. FP8 against INT4 is a quantization benchmark wearing an engine benchmark's clothes.

Hold maximum context identical. The context limit sets KV cache reservation, which sets achievable batch size, which sets throughput. Two engines with different context limits are not being compared.

Hold concurrency identical, at what you actually run. Numbers at batch size one, or at batch size 256 when your production concurrency is eight, tell you nothing. Sweep concurrency and plot the curve; the crossover points are the interesting part.

Use your own prompt distribution. Length distributions drive the prefill and decode balance, and the prefix overlap rate in your traffic decides whether prefix sharing helps you at all. A synthetic set with no shared prefixes hides SGLang's main advantage; one with a fixed prefix exaggerates it.

Separate prefill from decode. Report time to first token and inter-token latency rather than a blended tokens-per-second figure: prefill is compute bound, decode is memory bandwidth bound, and a change helping one often hurts the other.

Report percentiles at a stated request rate, not means, and warm up before measuring. Compiled engines, JIT paths, graph capture, and caches all need warm state to be representative, and a prefix cache will make a repeated run look spectacular for reasons that will not reproduce in production.

Operational realities

Memory tuning. Every engine exposes a knob for what fraction of accelerator memory it may claim, plus maximum context, maximum batch, and KV cache settings. These interact, and the failure mode is an out-of-memory error under peak load rather than at startup. Load test to your real peak before trusting a configuration.

Observability. You want per-request queue time, prefill and decode time, cache hit rate, running and waiting batch sizes, and KV cache utilisation as metrics. All three expose Prometheus-shaped metrics; the useful ones differ. Prefix cache hit rate tells you whether a prefix-sharing engine is earning its place, so confirm it is exported in the version you deploy.

Behaviour under overload. Find out what happens when the queue exceeds capacity: requests queue, get rejected, or preempt running sequences. Preemption means recomputing KV state later, throughput paid for twice. Put admission control in front of the engine rather than letting the scheduler discover your limits during an incident.

Upgrade cadence. All three release frequently. Pin versions, read changelogs for default changes, keep a rollback path. Shifting defaults have broken more deployments than removed features have.

The build step, if you compiled. TensorRT-LLM needs engine builds in CI, artifacts versioned alongside the model, a matrix if you run more than one GPU generation, and rebuilds on model, precision, parallelism, or driver-stack changes. Underestimating that system is the most common reason teams adopt it and then quietly migrate off.

Selection checklist

• Is the model set fixed, or will it change? Change favours a runtime engine.
• Is the hardware NVIDIA-only and staying that way? If not, portability is a hard constraint.
• What fraction of a typical request is a prefix shared with other requests? If it is high, prefix sharing is your largest available win.
• How long is your context, and is the model sparse? Both push you toward the projects doing active work there, and toward budgeting tuning time.
• Do you need constrained output, and how complex are the schemas at concurrency?
• What is your real concurrency, and what are your latency targets at p50 and p99?
• Can you own a build step in CI, artifact storage and GPU-generation matrix included?
• Who is on call, and how much operational novelty can they absorb mid-incident?

Still undecided: run vLLM, instrument it properly, collect a month of real traffic, and answer the questions against data. Starting with the portable default and migrating later costs far less than committing to a compiled, vendor-coupled deployment for a workload you have not measured.