Ternary Bonsai 2 27B: Architecture, Benchmark Reality, and Runtime Limits

On September 17, 2026, Prism ML released Ternary Bonsai 2 27B, a compressed multimodal language model derived from Qwen3.8 27B. By replacing conventional 16-bit floating-point weights with a ternary representation—where each parameter takes one of exactly three discrete values: -1, 0, or +1—the release compresses a 53.8 GB base model down to a 5.93 GB GGUF file.

The headline claim from Prism ML is that Bonsai 2 retains 98.2% of its full-precision parent's aggregate benchmark score while running at an effective 1.76 bits per weight. On paper, this enables a 27-billion-parameter model to fit comfortably within the unified memory of an entry-level laptop or a consumer GPU, delivering up to 142.5 tokens per second on an NVIDIA RTX 5090 and 46.8 tokens per second on an Apple M5 Max.

However, independent evaluations and community testing on hardware ranging from Apple Silicon to workstation Blackwell GPUs have begun to map the boundaries of this compression technique. While raw math, coding, and short-context instruction-following show remarkable resilience, long-horizon tool execution, agentic workflows, and real-world runtime stability tell a more nuanced story. Furthermore, running the model requires a proprietary fork of llama.cpp due to unmerged activation rotation routines, and third-party security researchers have already demonstrated runtime abliteration of the model's refusal guardrails.

The Architecture: Compressing Qwen3.8 27B Without Retraining

Ternary Bonsai 2 27B is not a newly trained foundation model. Prism ML did not execute a ground-up pretraining run; instead, it applied quantization-aware compression and custom low-bit kernel compilation to Alibaba's Qwen3.8 27B.

The underlying architecture retains the exact topology of Qwen3.8 27B:

• Parameter Distribution: 27.36 billion total parameters across 64 transformer blocks, with 24.35B in the language backbone, 2.54B in the embeddings and language model head, and 0.47B in the 27-block vision tower.
• Attention Mechanism: A hybrid-attention backbone featuring approximately 75% linear attention and 25% full softmax attention, paired with SwiGLU MLP blocks, Rotary Position Embeddings (RoPE), and RMSNorm.
• Context Window: Native 262,144 tokens (262K), supported by the linear attention layers which keep memory bandwidth and KV-cache footprint manageable at extended context lengths.
• Multimodal Tower: The vision encoder is packaged separately as a 4-bit mmproj component (~0.63 GB), loaded only when visual inputs are processed.

Prism ML's core technical contribution consists of two parts: the quantization-aware ternary transformation pipeline and the platform-specific low-bit kernels required to execute ternary arithmetic directly on Apple Silicon (Metal) and NVIDIA (CUDA) hardware without decompressing tensors back to FP16 in memory.

How Ternary g128 and Rotated Basis Work

To understand how a 27B model operates at 1.76 bits per weight, it is necessary to examine how ternary weights are structured and scaled.

The {-1, 0, +1} Discrete Space

In standard deep learning runtimes, weights are stored as 16-bit floating-point values (FP16 or BF16), offering 65,536 distinct numerical states per parameter. In a ternary network, the weight vocabulary is constrained to three values:

W ∈ {-1, 0, +1}

A naive encoding requires 2 bits per weight (which can address 4 states). However, pure ternary weights without scaling lack the dynamic range necessary for neural network convergence. Bonsai 2 resolves this by grouping parameters into blocks of 128 consecutive weights (g128), with each group sharing an FP16 scale factor:

w_i = s_g × t_i

Where:

• t_i ∈ {-1, 0, +1} is the ternary trit.
• s_g is the shared 16-bit floating-point scale factor for group g.

The inclusion of 0 as an explicit third state provides intrinsic sparsity, allowing blocks of weights to remain completely inactive when appropriate.

Walsh-Hadamard Orthogonal Rotation

Naive quantization below 4 bits per weight typically leads to catastrophic degradation because activation distributions in large language models exhibit extreme outlier channels. If a weight matrix with sharp numerical spikes is clipped to {-1, 0, +1}, critical representational capacity is lost.

To circumvent this, Prism ML applies an orthogonal rotation to the weight matrices prior to quantization:

1. The weight matrix is transformed blockwise using a Walsh-Hadamard matrix combined with a randomized diagonal sign matrix S ∈ {-1, +1} at block size 1024.
2. The rotation diffuses the energy of outlier channels across all coordinates, flattening the distribution into a uniform variance space where ternary quantization causes minimal distortion.
3. At inference time, the runtime must apply the inverse/matching transform to the incoming activations before multiplying by the ternary weights.

Because the rotation is folded directly into the stored weights during offline preparation, it introduces zero storage overhead. However, calculating the Hadamard transform on activations sits directly on the critical latency path of every projection at batch size 1. Prism ML addresses this by fusing the sign flips into the kernel load path on Metal and distributing the transform across thread blocks on CUDA.

Dissecting the Numbers: 1.585, 1.71, 1.72, and 1.76 Bits

Several bit-width numbers have circulated around Bonsai 2. Each measures a distinct aspect of the model:

The primary download artifact, Ternary-Bonsai-2-27B-PTQ1_0.gguf, measures exactly 5.947 GB on disk against the 53.808 GB FP16 baseline—confirming a 9.05x reduction in storage footprint.

The alternative packing format, PQ2_0 (7.25 GB, 2.16 bits/weight), stores each trit in a dedicated 2-bit container rather than packing trits across byte boundaries. While PTQ1_0 minimizes memory bandwidth traffic (favoring memory-bound architectures like mobile GPUs and older Ada Lovelace cards), PQ2_0 reduces arithmetic unpacking overhead, yielding higher prompt-processing throughput on compute-heavy architectures like NVIDIA Hopper, Blackwell, and Apple M-series chips.

Benchmark Analysis: Vendor Claims vs. Reality

Prism ML's headline 98.2% retention metric is based on a proprietary 20-benchmark evaluation suite comparing Bonsai 2 against the unquantized Qwen3.8 27B baseline (aggregate score of 83.9 vs. 85.4).

Figure 1: Benchmark retention across evaluation domains and runtime execution requirements for Bonsai 2 27B.

A granular inspection of the benchmark breakdown reveals significant performance variance across task domains:

Where Ternary Compression Succeeds

• Instruction Following (82.66 vs. 81.25): Bonsai 2 actually scored higher than the full-precision parent on instruction-following evaluations.
• Mathematics (96.57 vs. 97.06): On AIME26, Bonsai 2 recorded 95.83%, remaining within half a percentage point of full precision.
• Coding (81.58 vs. 82.17): On LiveCodeBench, the model achieved 90.07%, demonstrating that discrete weight states retain programmatic syntax and algorithmic reasoning effectively.

Where Ternary Compression Degrades

• Knowledge & Broad Reasoning (83.95 vs. 86.66): A 2.71-point deficit accounts for a substantial portion of the aggregate degradation.
• Vision Integration (78.59 vs. 81.64): Although the vision tower itself is uncompressed (4-bit mmproj), the language model's capacity to interpret multimodal projection tokens dropped by 3.05 points.
• Long-Horizon Engineering Tasks:
  • Terminal-Bench 2.1: Bonsai 2 scored 52.8 compared to 69.7 for full precision (a 24.2% drop).
  • SWE-bench Verified: Bonsai 2 scored 60.8 compared to 80.6 for full precision (a 24.5% drop).

While short-horizon reasoning and conversational code generation remain intact, multi-step agentic execution over extended command sequences suffers noticeable degradation. Compounding errors across iterative tool invocations expose the subtle loss in representational precision.

Quantization-Aware Training vs. Post-Training Quantization (IQ2_XXS)

The performance advantage of Bonsai 2 becomes clear when contrasted with standard Post-Training Quantization (PTQ). Conventional quantization approaches attempt to round or optimize existing FP16 weights without re-exposing the model to training loss.

Comparing Bonsai 2 against an IQ2_XXS GGUF build of Qwen3.8 27B illustrates the difference:

Standard post-training quantization at 2 bits (IQ2_XXS) degrades unevenly. While surface knowledge (MMLU-Redux) appears preserved at 85.79, complex reasoning cascades collapse—AIME26 drops by nearly 20 points, and GPQA Diamond falls to 65.45.

Because Bonsai 2 incorporated the ternary discretization and Hadamard rotation into its training and fine-tuning loops, it maintains deep reasoning pathways that naive rounding destroys.

Hardware Throughput and Operational Realities

Prism ML reported impressive hardware throughput benchmarks:

• NVIDIA RTX 5090: 142.5 tok/s decode on PQ2_0 (0.582 mWh per token).
• Apple M5 Max: 46.8 tok/s decode, 765 tok/s prompt processing.
• Apple M5 Pro: 27.7 tok/s decode (sustained 27.0 W GPU power draw).
• Apple M4 Pro: 18.0 tok/s decode.

Independent Testing and Caveats

Independent evaluations across developer forums and community testing revealed practical constraints:

1. Throughput Variance: On consumer hardware, actual throughput varied widely. While M5 Pro systems reached 44 tok/s under clean configurations, misconfigured Metal tensor environments resulted in speeds dropping to 20 tok/s, and entry-level M2 Mac Minis hovered around 7–8 tok/s.
2. Repetition Loops: Testers utilizing browser-based WebGPU and local runtimes noted that under specific open-ended prompts, the model occasionally entered repetitive generation loops—a behavioral artifact often linked to low-bit quantization in attention projection layers.
3. Proprietary Fork Dependency: Bonsai 2 cannot be run on stock llama.cpp. Upstream llama.cpp does not currently support the Hadamard activation rotation runtime. If a user attempts to load the GGUF file in mainline llama.cpp, the engine either rejects the tensor types or, worse, executes without rotation, producing fluent-looking syntactic gibberish. Users must compile Prism ML's dedicated fork.
4. MLX Limitations: The MLX repository (prism-ml/Ternary-Bonsai-2-27B-mlx-2bit) is tailored specifically for Apple Silicon Metal pipelines and contains no CUDA kernels. Linux and Windows users must rely exclusively on the GGUF CUDA fork.

Security Analysis: Runtime Abliteration and Guardrail Bypass

Shortly after Bonsai 2's release, security researchers at OrcaRouter published OrcaRouter Ternary Bonsai 2 27B Uncensored. This release highlights an important aspect of low-bit model security: runtime refusal abliteration.

Traditional model jailbreaking or uncensoring often requires fine-tuning model weights or applying weight deltas (e.g., orthogonal projection of refusal directions across weight matrices). In Bonsai 2, modifying individual weights directly is mathematically complex due to the discrete {-1, 0, +1} quantization grid and the Hadamard basis.

Instead, the uncensored variant operates entirely at the inference runtime layer:

• Researchers identified the activation direction associated with refusal behaviors across the intermediate residual streams.
• During generation, the runtime subtracts the refusal vector projection from the activation state in real time:

x_clean = x - (x · v_refusal) v_refusal

Because this intervention occurs strictly in activation memory, the underlying ternary weights remain bit-identical to the official release (Ternary-Bonsai-2-27B-PTQ1_0.gguf).

This demonstrates that low-bit quantization formats offer no intrinsic resistance to alignment stripping. Enterprises deploying quantized models locally cannot rely on frozen or proprietary low-bit weights as a tamper-proofing mechanism for safety policies.

Deployment Recommendations for Engineering Teams

For teams evaluating Ternary Bonsai 2 27B in production or edge environments:

1. Target Workloads Appropriately: Bonsai 2 is well-suited for single-turn coding assistance, structured data extraction, and constrained conversational tasks where local execution and low memory overhead are mandatory.
2. Avoid Autonomous Multi-Step Agents: Given the ~24% performance drop observed on SWE-bench Verified and Terminal-Bench 2.1, Bonsai 2 should not be deployed as an autonomous software-engineering agent where multi-step terminal or repository interactions compound errors.
3. Verify Runtime Compatibility: Ensure deployment pipelines are pinned to Prism ML's validated runtime forks. Monitor upstream pull requests to llama.cpp for native Hadamard kernel adoption before standardizing container images.
4. Enforce External Guardrails: Do not depend on model-native refusal training for security boundaries. If deploying locally to process untrusted inputs, implement external input/output validation layers rather than relying on internal alignment vectors.