MLX vs llama.cpp on Apple Silicon: Which Local LLM Runtime to Actually Use

Once you have decided which model to run on your Mac, the next question arrives immediately and gets far less careful treatment: what do you run it with. Two answers dominate, and nearly every desktop app, wrapper and tutorial in the local-LLM world sits on top of one of them. They are llama.cpp and MLX, and choosing between them is less about which is faster this month than about what each one was built to be.

Before either matters, it is worth being precise about what makes Apple Silicon an unusual host for this workload, because that determines your ceiling and no runtime choice moves it.

Unified memory changes the shape of the problem

On a conventional GPU box, weights live in VRAM: a separate, fixed, comparatively small pool you pay to copy data into across PCIe, and when a model does not fit you quantize harder, shard across cards, or give up. On an Apple Silicon SoC, the CPU and GPU address the same physical memory. There is no separate VRAM budget and no host-to-device transfer of weights, because there is no device to transfer to. A large-memory Mac can hold a model that would otherwise need several discrete accelerators wired together.

That is the structural advantage, and it is real. The constraint sits right next to it. Autoregressive decoding — emitting one token at a time — is bound by memory bandwidth rather than arithmetic. Each token requires streaming the active weights out of memory and through the compute units, and the matrix-vector shape of single-token generation gives the hardware very little to chew on per byte fetched. The ceiling on decode speed is therefore roughly the chip's memory bandwidth divided by the bytes of weight read per token, and in practice you land below that.

Which is why the bandwidth figure for your specific chip predicts token rate better than any other single number, core count included. Community testing reports place a 512 GB M5 Ultra at roughly 1200 GB/s of memory bandwidth; the same testing puts four DGX Spark units running tensor-parallel at TP=4 at roughly 1092 GB/s aggregate — four networked machines to land near one workstation on this particular axis. Bandwidth varies enormously across the Apple Silicon lineup, and the base, Pro, Max and Ultra tiers of a generation are nowhere near each other, so look up the number for the chip you actually own.

Prefill behaves differently: processing the prompt is batched and compute-bound, which is where GPU throughput and kernel quality matter. Two phases, two bottlenecks, which is why you measure them separately.

These are not competitors in the usual sense

The framing that makes everything else fall into place is that llama.cpp and MLX are not two implementations of the same idea.

llama.cpp is a portable inference engine, written in C and C++ with no heavy runtime dependency, targeting a long list of hardware backends. Apple Silicon via Metal is one platform it supports extremely well. Its purpose is to run transformer models for inference, anywhere.

MLX is an array framework for Apple Silicon, built by Apple's machine learning research group. It occupies roughly the slot NumPy or PyTorch occupies: arrays, operations on arrays, automatic differentiation, a device model. Running a language model is one application of it, not its definition.

Nearly every practical difference below is downstream of that. One is an engine with a narrow job and enormous reach; the other is a framework with a broad job and a single platform.

llama.cpp: the default, and the safe choice

The case starts with distribution. Models ship as GGUF files: one self-contained file carrying the weights, the tokenizer, and the metadata the runtime needs to configure itself. You download a file and run it — no environment to assemble, no matching set of Python package versions, no config directory to keep in sync with the weights. For anyone who has lost an afternoon to a dependency conflict, that is not a small thing.

Quantization is where the project has done its deepest work. The K-quant family is a graded ladder of size-versus-quality tradeoffs, with mixed precision applied per tensor so the layers most sensitive to quantization keep more bits than the ones that tolerate it. Below those sit lower-bit importance-matrix variants, calibrated against sample data to decide where precision is worth spending. The effect is many rungs between "full quality, does not fit" and "fits, badly damaged", and you can usually find one that lands where you need it.

The Metal backend is mature and well-tuned. Layer offload is configurable: you choose how many transformer layers run on the GPU and leave the rest on CPU, which lets a model that slightly exceeds your GPU budget still run, slowly, rather than not at all.

Then the ecosystem, which is usually the argument that decides it. Ollama, LM Studio and a long list of desktop and server applications are built on or ship llama.cpp. The project ships its own server with an OpenAI-compatible endpoint, so anything speaking that API points at it unmodified. Model coverage is exceptionally wide, and support for genuinely new architectures tends to land there first — a GGUF conversion of an unfamiliar new model usually exists within days.

Portability is the last piece and the most underrated. The same GGUF file runs on your Mac, on a Linux server with an NVIDIA card, and on a Windows laptop. If you develop against a model locally and deploy it elsewhere, or support colleagues on other operating systems, that removes an entire class of problem.

The weaknesses are exactly where its design predicts. It is an inference engine: training and fine-tuning are not its domain, and you should not plan a workflow around doing them there. Customization means writing C++ against its internals, which is reasonable if that is your language and painful if it is not. And the abstraction is a model runner, not a toolkit, so if what you want to build does not look like "load model, generate tokens", you are working against the grain.

MLX: Apple Silicon as a first-class target

MLX starts from the opposite end. The API is NumPy-like and immediately familiar to anyone who writes Python for numerical work, with a design borrowed from PyTorch, Jax and ArrayFire. Function transformations compose — gradients, vectorization and compilation can be applied to each other. Evaluation is lazy: operations build a graph and materialize only when a result is needed, which lets the framework fuse work and skip computation nothing consumes.

The unified memory design is what reads as genuinely different if you come from CUDA. Arrays live in shared memory rather than a device-specific allocation. You do not move data between CPU and GPU; you specify where an operation runs, not where the data lives. Code that mixes the two accumulates no transfer calls, and a whole category of device-placement bugs does not exist.

The decisive difference is training. MLX supports training and fine-tuning, not just inference, so LoRA work on a Mac is native rather than a workaround: the mlx-lm package ships LoRA and QLoRA fine-tuning as a first-class entry point alongside generation and a server. That single fact settles a lot of otherwise-close decisions. If your loop is "fine-tune, evaluate, adjust the data, fine-tune again" and you want it running on the machine in front of you, MLX is the answer and llama.cpp is not a candidate.

mlx-lm handles the model plumbing too: it converts from the standard open-weight safetensors format and quantizes during conversion with configurable bit width and group size, and the community publishes a large body of already-converted models. The format story is not a separate universe, just a conversion step usually done for you.

The tradeoffs deserve stating without softening. MLX is Apple-only by construction — not an oversight to be fixed later but the premise, and nothing you build on it runs on a Linux GPU box. The ecosystem is younger and moves faster, so APIs shift more than a mature project's would and you occasionally read source to answer what the docs have not caught up to. And there are fewer polished desktop applications, though some now ship both backends and let you pick per model.

Side by side

On performance, and why there are no numbers here

Relative speed shifts between releases and depends on the model, the quantization scheme, the context length and the chip. Any tokens-per-second table printed today would be misleading within a couple of release cycles and would not describe your machine anyway. What stays true is the set of factors governing throughput:

Memory bandwidth of your chip. The dominant term for decode, a property of the silicon you bought, and nothing in software changes it.

Model size and quantization. Fewer bytes per parameter means fewer bytes streamed per token, which is why a 4-bit model decodes substantially faster than the same model at 8-bit on identical hardware. This is the main lever you control.

Mixture-of-experts routing. Decode reads only the active parameters per token, not the total — GLM-5.3-Flash is 320B total parameters with 18B active — so decode is far cheaper than the headline size suggests. It does not reduce memory occupancy: all of it stays resident, because any expert may be selected.

Context length. The KV cache grows linearly with context and is read on every step, so a long conversation is slower than a short one on the same model, and the cache competes with the weights for both capacity and bandwidth.

Whether the model fits in physical memory. Not a gradient, a cliff. Stay inside memory and the above holds. Exceed it, macOS starts compressing and swapping, throughput collapses by an order of magnitude or more, and no runtime choice rescues you. The fix is a smaller model or harder quantization, not a different engine.

Benchmark them yourself, properly

The only number that matters comes from your machine and your model. Both projects ship tooling for it: llama.cpp includes a dedicated benchmarking binary, and mlx-lm reports timing from its generation entry point.

Hold these constant across runs or you are measuring noise:

• Same model at the same effective quantization. A 4-bit GGUF against an 8-bit MLX conversion tells you about bit width, not runtimes.
• Same context length and the same prompt. Reuse one fixed prompt for both.
• Warm start. Discard the first run; weights paging in from disk is a filesystem cost, not an engine cost.
• Nothing else heavy running. Unified memory is shared with everything open, browser included.

Then report prefill and decode separately. A runtime can win one and lose the other, and a blended tokens-per-second figure hides precisely the information you need. Run each configuration several times and take the median.

Memory sizing, which is the real constraint

Installed memory is not usable model memory. The OS, the window server, your browser and everything else open draw from the same pool, and macOS separately limits how much memory may be wired for GPU use. That wired limit governs how large a model the GPU can hold. It is exposed as a tunable and people do raise it, but the default exists to keep the rest of the system functional, and pushing it too far trades a working Mac for a marginally larger model.

Plan from a budget rather than from constants: start at installed memory, subtract a realistic allowance for the OS and your working set, then subtract the KV cache for the context length you actually intend to use. What remains is your weight budget, and quantization has to bring the model under it.

The verified model sizes make the stakes concrete. GLM-5.3-Flash at FP8 is approximately 328 GB of weights; Qwen3.8-Flash at NVFP4 is approximately 135 GB. Both are out of reach for every Mac configuration except the very largest, and the 328 GB figure is out of reach even for most of those once you account for the OS and a usable context. That is the whole point: quantization choice decides what a given Mac can run, and it is a more consequential decision than the choice of runtime.

Which one should you use

Run models with minimal setup, or want a desktop app. llama.cpp, almost certainly through a wrapper rather than directly. Download a GGUF, point the app at it, done.

Need the same model files on Linux or Windows too. llama.cpp. Not close: GGUF is portable and MLX is not.

Want to fine-tune or train on the Mac. MLX. LoRA and QLoRA are supported paths there, and llama.cpp does not compete for this work.

Writing Python and building something custom on Apple Silicon. MLX. You get a real array framework with autodiff and composable transformations, not an engine driven from outside.

Running very new or unusual architectures. llama.cpp, generally first to support them. Breadth of coverage is where ecosystem size pays off most directly.

Serving many concurrent users. Neither, honestly. The llama.cpp server handles a handful of parallel slots and will do for a small team, but neither project targets throughput-oriented serving the way a dedicated engine on a Linux box does, and a Mac is not the right host for it.

They are not exclusive

Nothing forces a single choice. Models convert between formats, the two install independently without conflict, and plenty of people run both: llama.cpp for everyday inference, because it is stable and every tool talks to it, and MLX for fine-tuning experiments and anything custom.

The reasonable default is to start with llama.cpp, because it runs almost anything and the ecosystem carries you. Reach for MLX when you want to train something, or when you are writing the code yourself and want the Mac to be a first-class target rather than a supported platform.

Checklist

• Just run models, minimal fuss, or want a GUI — llama.cpp, via a wrapper
• Need the same files on Linux or Windows — llama.cpp
• Fine-tuning or training on the Mac — MLX
• Custom Python work on Apple Silicon — MLX
• Brand-new or exotic architectures — llama.cpp
• Many concurrent users — neither; a Linux box with a dedicated serving engine
• Not sure — llama.cpp first, add MLX when you need training