Tag archive

Inference Engineering

Coverage tagged inference engineering.

Infographic mapping one to four NVIDIA DGX Sparks to the recommended local model at each node count: Qwen3.8 Flash Next, DeepSeek V4 Flash, GLM-5.3 Flash, DeepSeek V4.1 Flash.

Inference Engineering: A Practical Primer for Serving Your Own Models

Prefill is compute-bound, decode is bandwidth-bound, and KV cache is what really caps concurrency. The metrics, memory maths and tradeoffs behind serving.

Related desks

Research

Latest

Sep 12, 2026, 12:37 PM EDT · Research · Mlx12 minMLX vs llama.cpp on Apple Silicon: Which Local LLM Runtime to Actually Use

One is a portable inference engine, the other an array framework that also trains — and that single difference settles almost every practical question.

Sep 12, 2026, 12:37 PM EDT · Operations · Llm Evaluation14 minHow to Build an LLM Evaluation Harness That Catches Real Regressions

Public benchmarks only shortlist a model; your own cases decide whether to ship it. Here is the dataset, the scorers and the CI gates that catch real drift.

Sep 12, 2026, 12:37 PM EDT · Research · Prompt Caching11 minPrompt Caching Explained: Cutting Agent Costs Without Touching the Model

Your agent re-sends the same long prefix on every step and pays to recompute it. Here is the KV mechanism, the ordering rule it forces, and the break-even math.

Sep 12, 2026, 12:37 PM EDT · Research · Speculative Decoding11 minSpeculative Decoding and Multi-Token Prediction: How LLMs Generate Faster

Decode is memory-bound, so checking several tokens in one pass is nearly free. Here is the mechanism, the arithmetic behind it, and why quality never moves.