PrismML Stuffs a 27B Model into Your iPhone — With Barely Any IQ Drop

Base is still Qwen3.6-27B; Bonsai packs the whole net into extreme bits

The base is Alibaba Qwen team's Qwen3.6-27B: about 27B parameters, hybrid attention, roughly 75% linear-attention layers and 25% full attention, with native long-context support. PrismML did not train a brand-new 27B from scratch; it mapped this already-trained net to binary or ternary weights, then shipped custom kernels that actually run those weights.

Vision tower is separate ~4-bit, ~0.63GB on disk; usually not resident for pure text. Optional DSpark draft-accel layer adds roughly another 1.8–2.0GB.

What blocks local 27B first: "every generated token reloads the whole body"

In local token-by-token generation, the device almost has to stream the full model weights from memory again for every new slice of text. In small-batch settings, the bottleneck is usually memory bandwidth, not peak FLOPs. Bigger model → more data moved per step → speed, heat, and battery all suffer together.

The numbers are blunt: FP16 Qwen3.6-27B is ~54GB. A common "4-bit" pack Q4_K_XL averages ~5.2 bit/weight in reality, ~17.6GB. More aggressive IQ2_XXS is labeled "2-bit" but is really ~2.8 bit, ~9.4GB—still too heavy for most phone single-app budgets, and long-chain reasoning scores collapse hard.

That is the real point of "what's so special about 27B": not that the parameter count is novel, but that this capability tier was first squeezed inside the physical envelope of on-device deploy. Ordinary methods still hold scores above 4-bit; further down, the drop is often not uniform—tool-call parsing and multi-step coherence start to shatter. Bonsai's claim: in the sub-2-bit zone it still keeps a slice of those behaviors.

You compress weights; long context is gated by cache

Qwen3.6-27B's hybrid attention helps a lot: of 64 layers, only 16 full-attention layers grow cache with length; FP16 cache is about ~64KB/step, roughly 4× cheaper than a full-attention 27B. Even so, 262K-context FP16 cache is still ~17.2GB—larger than the 1-bit weights themselves.

Bonsai weights themselves are quite "tolerant" of 4-bit KV cache. Measured with output-distribution forward-KL: with the same 4-bit cache on, Bonsai's deviation from its own FP16-cache baseline is about an order of magnitude lower than FP16 / common 4-bit weights. In other words, compressed weights and compressed cache pull the same direction on this path—they do not sabotage each other. Project docs still mark 4-bit KV as experimental: mainly memory savings, slightly slower decode; calibrate on real workloads.

Average holds ~90%; losses land on instruction, tools, and vision

Evals ran in thinking mode across 15 tasks covering knowledge, math, coding, instruction following, tool use, and vision. FP16 baseline average 85.07; ternary 80.49; 1-bit 76.11. The stark contrast is not how many points Bonsai lost, but how ordinary aggressive quant collapses: IQ2_XXS drops AIME26 from 93.33 to 57.5 and LiveCodeBench from 87.77 to 56.4; on the same suite, 1-bit Bonsai stays above 87 on AIME and 76.4 on LiveCodeBench.

Positioning is therefore clear: ternary fits memory-rich laptops when you want close to stock 27B feel; 1-bit first solves "it fits"—do not read "~90% average" as "~90% agent ability." The site says tool use is only a few points below full precision; that roughly holds for ternary; for 1-bit, tool-category average gap is near 14 points.

Official "intelligence density per GB" stresses volume efficiency: 1-bit ~0.53/GB, over 10× FP16 and ~2.7× the densest common low-bit pack. This metric is PrismML-defined; the denominator is raw size, so it heavily rewards smaller models. Good for deploy efficiency, not a substitute for absolute capability scores. Source: PrismML launch page.

Where the eval framing needs a discount

EvalScope, vLLM, H100, per-task generation length budgets, and scoring methods are documented—more complete than a radar-chart-only press kit. Third-party reproduction is not yet public. Most items ran once; Bonsai sampling temperature 0.7 vs Qwen baseline and common quants at 1.0—generation settings are not fully aligned. Overall average is a simple mean of 15 tasks; math is 4 of them, and Bonsai happens to retain best on math, so "95% / 90%" looks better than real product loads that lean tool- or instruction-heavy. People building agents should prioritize BFCL v3, τ²-Bench, IFEval, IFBench over the overall average alone.

Speed has two layers: low-bit movement, and draft accel on CUDA

Low-bit weights directly cut bytes moved each step, so token-by-token generation benefits most. Speed is reported uniformly as tg128 (~128 generation units) and pp512 (process 512 input units), batch size 1, excluding draft layer and vision tower.

DSpark speculative decoding: a small draft model guesses a segment first (depth k=4); the target verifies in one pass. Verification is lossless—output distribution matches target-only. On H100, ternary rises from 98.0 to 131.8 tok/s (~1.34×); 1-bit from 104.8 to 143.8 (~1.37×). On Apple Silicon single-request paths, multi-row verification does not amortize cost, so it is off by default; the project also marks it highly experimental.

Energy: on M5 Pro, 1-bit is ~0.275 mWh/token, an order of magnitude under six data-center GPUs at 0.63–1.32 mWh/token. On iPhone 17 Pro Max, about 672 tokens per 1% battery, sustained ~10.8 tok/s; after ~5.2 minutes and ~3360 tokens, battery goes 100%→95% with mild thermal throttle. Vendor self-tests, but the order of magnitude is useful.

Worth trying first—and what not to assume

Desktop is easiest; phone is still a developer path

Bottom line: if you want to touch it now, install the official desktop demo first. On phone, official work proves it runs, but there is no App Store one-tap finished app yet—you integrate 1-bit weights + Apple-side MLX / Swift kernels yourself, or wait for an official/third-party wrapper.

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo

# 默认：三元 27B（笔记本更合适）
export BONSAI_MODEL=27B
# 若 27B 仓库仍私有，填你的 HF 读 token
export BONSAI_TOKEN="hf_your_token_here"
./setup.sh
./scripts/start_llama_server.sh
# 浏览器打开 http://localhost:8080

# 改成 1-bit 手机同款家族：
# BONSAI_FAMILY=bonsai BONSAI_MODEL=27B ./setup.sh

Numbers and demos come from the launch materials and technical whitepaper; third-party reproduction and mid-range phone fitness still need your own checks. 27B weights first push local deploy near ~4GB; long context, multimodal residency, and agent reliability still need scenario-level testing.