Does Bonsai 2 27B really keep 98% of Qwen3.8 in 5.95 GB?

We've got a 24 GB card in the office box and a 27B model we'd like to run on it with room left for a real context. Until this week the honest options were a 4-bit GGUF at 17.6 GB or a "2-bit" one that scored like an 8B. On 17 September 2026 PrismML released Ternary Bonsai 2 27B, a version of Qwen3.8 27B where every weight is -1, 0 or +1, and the file is 5.95 GB. Apache 2.0, 262K context, vision tower optional. The headline is 98.2% of the full-precision score, and that number is real. So is the one they keep outside the average.

What 1.72 bits per weight buys you

Ternary means each weight stores one of three values, and log2(3) is about 1.585 bits. Add one FP16 scale shared by every group of 128 weights and you land at 1.71. A sliver of tensors (26.2 million parameters, under 0.1%) stays in higher precision for the recurrent state path and the norms, which brings the whole model to 1.72. Real kernels need a packed layout, so there are two GGUF files. PTQ1_0 packs the trits densely at 1.75 bits per weight and 5.95 GB. PQ2_0 gives each trit a 2-bit slot, costs 7.21 GB, and is cheaper to unpack. The optional vision tower is a separate 0.63 GB mmproj in Q8_0 that only loads when you send an image.

That's the weights. Not the memory. The KV cache for a 262K context sits on top, and the base is a hybrid with roughly 75% linear-attention layers, which is what makes long context survivable at all on a laptop. PrismML's own line is "a 16 GB laptop or a single 24 GB GPU", and honestly that matches what we'd expect: a 7.21 GB file plus a 32K context fits a 4090 with plenty spare, and the Apple numbers were measured on M5 Pro and M5 Max machines running the 7.2 GB pack. The base is the same Qwen3.8-27B that shipped under Apache 2.0 in August, so the licence carries through cleanly.

Speed is bandwidth-bound at batch 1, and the model card's llama-bench table is refreshingly specific. RTX 5090: 129.9 tokens a second on PQ2_0, 120.5 on PTQ1_0. RTX 4090: 81.2 and 91.1. H100: 113.9 and 86.9. An L4 at 72 W: about 30 either way. M5 Pro laptop: 28.1 tokens a second on Metal, drawing 27.5 W on the GPU rail, against 300 to 455 W of board power on the Nvidia cards. The launch post says "up to 143 tokens a second" on the 5090, which is higher than anything in the model card table, and both numbers are theirs. I'd plan around the table. The smaller pack isn't always the faster one either: dense trits cost arithmetic to unpack, so PTQ1_0 wins on the 4090, the Ada 6000 and the L4, and loses on Hopper, Ampere and Blackwell where batch-1 decode isn't starved for memory. On a 5090 or an H100, download the 7.21 GB file.

Two benchmark suites, and only one in the headline

The 98.2% comes from an average over 14 thinking-mode benchmarks on the model card, or 20 in the whitepaper. Same retention figure both ways. The card's version reads: FP16 86.32, the 4-bit UD-Q4_K_XL build at 17.6 GB scores 85.18, a conventional IQ2_XXS at 9.4 GB scores 72.59, and Bonsai 2 at 5.9 GB scores 84.78. The per-benchmark rows show where the conventional 2-bit build dies: 57.5 on AIME26 and 56.4 on LiveCodeBench, while still posting 88.93 on MMLU-Redux, which is exactly why a quick chat test wouldn't catch it. Bonsai 2 posts 95.83 and 90.07 on those two, level with or above FP16. Math loses half a point (96.57 against 97.06 by category). Coding is flat. Instruction following is actually ahead.

Where it drops is the knowledge and vision rows. MuSR falls from 79.63 to 70.63, MMLU-Redux from 91.46 to 89.09, MMMU-Pro from 81.73 to 75.49 and OCR Bench v2 from 60.99 to 56.88. That pattern (reasoning intact, recall and perception softer) is consistent with a model that lost bits rather than structure, and it's the same shape PrismML reported on Bonsai 1 two months ago, when the ternary variant kept about 95%.

Then there's the pair the whitepaper reports but leaves out of the average, and MarkTechPost was right to pull them up. Terminal-Bench 2.1: 52.8 against 69.7 for the parent. SWE-bench Verified: 60.8 against 80.6. That's about 75% retained, not 98%. Long-horizon agent tasks compound small errors across dozens of tool calls, so a model that's 2% worse per step doesn't come out 2% worse at the end. PrismML's launch copy talks up "long-horizon agentic performance" and demos the model driving a coding agent on a 5090. Both things can be true. PrismML says it's better than Bonsai 1 at this, and it's still 20 points behind the model it was made from. For a local coding agent, that's the number to weigh, and it's the one we'd want to see someone outside PrismML measure. One tester on GitHub who ran the v1 model through an agent suite wrote that the claimed lead "did not hold at suite level" on wall time, and has a v2 run queued. We'd read that before anything else.

Running it, which means a fork

Stock llama.cpp rejects PTQ1_0 and PQ2_0 as unknown tensor types. Worse, the card warns it'll load a Q2_0 file without complaint and produce garbage, because the ternary path relies on a Hadamard rotation applied at runtime that mainline doesn't have. So you build or download PrismML-Eng/llama.cpp, and on Apple you use their MLX fork instead. There's a prebuilt release archive per platform if you'd rather not compile.

Drop -DGGML_CUDA=ON on macOS, Metal is the default there. -ngl 99 puts every layer on the GPU and -c goes up to 262144 if you've got the memory for the cache. It's a reasoning model and it thinks by default, so budget output tokens accordingly. The sampling flags are the ones PrismML recommends, and the repo they call the source of truth is PrismML-Eng/Bonsai-demo, which pins a known-good binary and covers the server, tool calling and image input with the mmproj file.

The fork is the part we'd flag to anyone running a fleet. A vendor fork of llama.cpp means you're on their release cadence for kernel fixes, and every upstream feature lands on their schedule or not at all. That's fine for a workstation. For a serving box it's a dependency you didn't have with a standard GGUF of a Qwen model. PrismML says returning the footprint advantage as latency on every GPU is "an active engineering target", which is a candid way of saying the kernels aren't done. I might be wrong about how long that takes, but ternary kernels in mainline llama.cpp would change the calculus completely, and nothing on either repo says it's coming.

Sources

PrismML, Introducing Bonsai 2 27B, 17 September 2026 (the release date, the 98.2% and 83.9 against 85.4 figures, the "up to 143 tokens/second" claim, Apache 2.0, the 95% retention of Bonsai 1). Hugging Face, prism-ml/Ternary-Bonsai-2-27B-gguf model card, read 20 September 2026 (the 1.72, 1.75 and 2.13 bits per weight, the 5.95 and 7.21 GB packs, the 0.63 GB vision tower, the 14-benchmark table and per-benchmark rows, the llama-bench throughput and power table, the fork requirement and the run commands). PrismML, Bonsai 2 27B whitepaper and PrismML-Eng/llama.cpp, September 2026 (the 20-benchmark average, the 26.2M high-precision parameters, the packings). MarkTechPost, PrismML releases Ternary Bonsai 2 27B, 18 September 2026 (the Terminal-Bench 2.1 and SWE-bench Verified rows from the whitepaper, the "16 GB laptop or a single 24 GB GPU" line, the Cline demo). GitHub, evanwtf/local-llm issue 479, September 2026 (an independent tester's note on the v1 agent results and a planned v2 run).

Frequently asked questions