Topic

Local Llm Inference

Coverage tagged local llm inference.

Articles: 3
Desk: 1

Data graphic: a large 3 GB figure labelled as OpenBMB's vendor claim for MiniCPM-V 4.7 running at 13+ tokens per second on an iPhone

MiniCPM-V 4.7: OpenBMB Claims a 35B Vision MoE Runs at 13 Tokens a Second in 3 GB on an iPhone

OpenBMB says its 35B-A3B vision MoE hits 13+ tok/s in about 3 GB on an iPhone 17 Pro Max. No quantisation is stated; the Q4_K_M GGUF is 21.2 GB.

Oct 14 · 4 min

Data graphic: 4-bit model file sizes against the memory budget for local LLMs on 16GB hardware. A 16GB Mac allows about 10.7GB of GPU memory by default, against roughly 15GB on a 16GB NVIDIA card, so Laguna XS 2.1 20.3GB and Qwen3.8-27B 16.8GB do not fit a 16GB card, while Devstral Small 2 24B 14.3GB , Gemma 4 26B A4B 14.2GB and gpt-oss-20b 12.8GB do. Practical context is 32K–64K for gpt-oss-20b, 8K–16K for Gemma 4 26B A4B and 4K–8K for Devstral Small 2 on the card, and 32K for Qwen3.5-9B ~5.6GB and 16K–32K for Gemma 4 12B 6.98GB on the Mac. The RTX 5060 Ti has 448GB/s of bandwidth to the RTX 4060 Ti's 288GB/s, and the verdict is to run gpt-oss-20b or Gemma 4 26B A4B on a 16GB card and Qwen3.5-9B on a 16GB Mac.

watchResearch Local Llm Inference

Best Local LLM for 16GB in 2026: RTX 5060 Ti, 4060 Ti and Mac mini M4 (Coding First)

gpt-oss-20b and Gemma 4 26B A4B lead on 16GB NVIDIA cards, while a 16GB Mac mini M4 is better served by Qwen3.5-9B. File sizes, context limits and VRAM math.

Sep 24 · 13 min

watchResearch Local Llm Inference

The RTX 5090 (32GB) Local LLM Deployment Blueprint: NVFP4 vs. FP8, SGLang vs. vLLM, and Context Limits

Can the RTX 5090 run 70B models? We break down 32GB VRAM sizing, Blackwell NVFP4 vs FP8 benchmarks, KV cache math, and vLLM vs SGLang configurations.

Sep 19 · 8 min