Owen Park

Prism ML’s Ternary Bonsai 2 27B compresses Qwen3.8 into 5.9GB at 1.76 bits per weight. Here is how the math works and what independent testing reveals.

One is a portable inference engine, the other an array framework that also trains — and that single difference settles almost every practical question.

Your agent re-sends the same long prefix on every step and pays to recompute it. Here is the KV mechanism, the ordering rule it forces, and the break-even math.

Decode is memory-bound, so checking several tokens in one pass is nearly free. Here is the mechanism, the arithmetic behind it, and why quality never moves.

Prompting for JSON gets you high-90s reliability; a grammar-driven token mask gets you a guarantee. The mechanism, the real costs, and where it still fails.

Your retriever found the right chunk and buried it at rank 31. A cross-encoder second pass fixes ordering without touching recall, and here is how to size it.

Dimension count, sequence length and model size decide your index memory, chunk limits and ingest hours. A framework that prices the retrieval quality you buy.

Most retrieval bugs are split bugs. A working reference on overlap, structure-aware and small-to-big splitting, with concrete defaults you can tune from.