Qwen3.6-27B is the base; Bonsai maps the whole network into extreme low-bit

The base is Alibaba Qwen's Qwen3.6-27B: roughly 27B parameters, hybrid attention (75% linear attention layers, 25% full attention), with native support for very long contexts. PrismML didn't train a bespoke 27B from scratch. Instead, they mapped this pre-trained network onto binary or ternary weights and wrote custom kernels to run those weights directly.

The vision tower sits at ~4-bit on disk (~0.63GB) and usually isn't kept in memory during text-only sessions. The optional DSpark speculative decoding layer adds another ~1.8–2.0GB.

The real barrier to a local 27B: you have to stream the entire model for every token

When generating text locally, the device has to re-read the entire set of model weights from memory for each new token. In low-batch scenarios, memory bandwidth — not raw compute — is usually the bottleneck. A bigger model means more data shuffled per step, which hurts speed, thermals, and battery life all at once.

FP16 Qwen3.6-27B is about 54GB. A typical "4-bit" package like Q4_K_XL actually averages ~5.2 bit/weight, landing around 17.6GB. The more aggressive IQ2_XXS is labeled "2-bit" but averages ~2.8 bit, ~9.4GB — still too heavy for most phone app budgets, and its long-chain reasoning scores collapse.

This is where "what's the big deal about 27B" actually lands: it's not the parameter count that's new, it's that this capability tier has been squeezed into a size that's truly deployable on-device. Regular methods hold up above 4-bit, but below that, scores don't just degrade evenly — tool-call parsing and multi-step reasoning coherence start to shatter. Bonsai's claim is that it keeps these behaviors largely intact even in the sub-2-bit zone.

It compresses weights, but the KV cache determines whether long contexts fit

Qwen3.6-27B's hybrid attention helps a lot: only 16 of the 64 layers have full attention caches that grow with sequence length. An FP16 cache costs about 0.064GB per 1K tokens — already roughly 4× cheaper than a full-attention 27B. Still, an FP16 cache at 262K context is about 17.2GB, which is larger than the 1-bit weights themselves.

Bonsai weights are quite "tolerant" of a 4-bit KV cache. Measured by forward-KL divergence from the output distribution: with the same 4-bit cache enabled, Bonsai's deviation from its own FP16-cache baseline is about an order of magnitude smaller than for FP16 or typical 4-bit weights. Compressing the weights and compressing the cache work together on this path, not against each other. The repo still flags 4-bit KV as experimental: it saves memory, decoding gets a bit slower, and you should calibrate it against your actual workload.

Average holds at ~90%; losses concentrate in instruction-following, tools, and vision

Benchmarks were run in thinking mode, 15 tests total, covering knowledge, math, coding, instruction following, tool calling, and vision. FP16 baseline averages 85.07; ternary gets 80.49; 1-bit scores 76.11. The most striking contrast isn't how much Bonsai drops — it's how badly typical aggressive quantization collapses: IQ2_XXS falls from 93.33 to 57.5 on AIME26 and from 87.77 to 56.4 on LiveCodeBench. On the same tasks, 1-bit Bonsai still scores above 87 on AIME and 76.4 on LiveCodeBench.

That's the split between the two versions: ternary is for laptops with enough RAM, aiming to stay close to the original 27B feel; 1-bit solves "it fits" first. Don't read "90% average" as "90% agent capability." A few points off on tool calling is plausible for ternary; for 1-bit, the tool category is nearly 14 points behind.

"Intelligence per GB" measures size efficiency: 1-bit hits ~0.53/GB, roughly 10× FP16 and about 2.7× the densest typical low-bit package. This metric is custom to PrismML, dividing raw scores by size, which strongly rewards smaller models. It's useful for deployment efficiency, not a substitute for absolute capability scores.

Where to discount the eval

They used EvalScope, vLLM, and H100, and published token budget and scoring methodology for each test — far more complete than a press release with only a radar chart. But no third-party replication exists yet. Most runs are single-shot; Bonsai used sampling temperature 0.7 while Qwen baselines and typical quantization used 1.0, so generation settings aren't fully consistent. The overall average is a simple mean of 15 tests, with math making up 4 of them — Bonsai happens to retain math best, so the "95% / 90%" figures look better than a tool-heavy or instruction-heavy real-world load would be. For agent work, look at BFCL v3, τ²-Bench, IFEval, and IFBench rather than the overall average.

Speed has two layers: low-bit data movement, and draft acceleration on CUDA

Low-bit weights directly reduce the bytes shuffled each step, so token generation benefits the most. Speeds are reported uniformly at tg128 (~128 tokens generated) and pp512 (512 input tokens processed), batch size 1, without draft layer and vision tower.

DSpark speculative decoding: a small draft model guesses a few tokens (depth k=4), then the target model validates them in one pass. Validation is lossless — the output distribution matches running the target model alone. On H100, ternary goes from 98.0 to 131.8 tok/s (~1.34×), and 1-bit from 104.8 to 143.8 (~1.37×). On Apple Silicon, with single-request workloads, multi-token validation can't amortize its cost, so it's off by default; the project itself labels it highly experimental.

On energy: 1-bit on M5 Pro draws about 0.275 mWh/token — an order of magnitude lower than the 0.63–1.32 mWh/token range across six data-center GPUs. On iPhone 17 Pro Max, it generates roughly 672 tokens per 1% battery, sustaining ~10.8 tok/s, and after ~5.2 minutes (~3,360 tokens) battery drops from 100% to 95% with mild thermal throttling. Treat these as ballpark figures; specific devices will vary.

What to try first — and what not to assume

Desktop is easiest; mobile is still a developer path

Want to try it now? The desktop demo is the smoothest route. It does run on phones, but there's no "App Store one-click install" app yet — you'd need to integrate the 1-bit weights with Apple's MLX / Swift kernels yourself, or wait for a future app wrapper.

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo

# Default: ternary 27B (better for laptops)
export BONSAI_MODEL=27B
# Only if the 27B repo is still private — add your HF read token
export BONSAI_TOKEN="hf_your_token_here"
./setup.sh
./scripts/start_llama_server.sh
# Open http://localhost:8080 in your browser

# To switch to the 1-bit phone-class family:
# BONSAI_FAMILY=bonsai BONSAI_MODEL=27B ./setup.sh

Data and demos above come from launch materials and the technical whitepaper. Independent replication and mid-range phone viability are still open questions. The 27B weights have pushed the local deployment bar down to ~4GB for the first time; long-context, multimodal residency, and agent reliability still need real-world testing per scenario.