AI startup PrismML released Bonsai 2 27B on September 17, the latest version of its ternary-quantized model family. The model is based on Alibaba’s open-source model Qwen3.8 27B. By compressing model weights to only three values—+1, -1, and 0 (i.e., “ternary weights”)—it shrinks a model that originally required about 54 GB of memory to 5.9 GB, reducing its size by about 9x, while retaining 98.2% of the original performance in comprehensive benchmarks. This means a 27B-class inference model can now run on personal computers and even high-end smartphones. For developers and users who have been waiting for “on-device AI” to mature, this is a key technological breakthrough.
Compression technology that came out of a Caltech lab
PrismML was founded by a research team from Caltech. CEO Babak Hassibi is a professor there and specializes in compression technology. The company’s advisory lineup is equally impressive: Ion Stoica is a co-founder of Databricks and director of UC Berkeley’s Sky Computing Lab, which has incubated well-known technology projects and startups such as Letta and SGLang. Investors include Khosla Ventures, Cerberus Capital, and Caltech itself.
The working principle of ternary weights
A typical large language model requires 16 bits to store each weight. PrismML’s approach simplifies each weight to three possible values: +1, -1, or 0. This “ternary” representation requires only log₂(3) ≈ 1.585 bits per weight. Adding a shared FP16 scaling factor for every 128 weights (an extra 16/128 bits) brings the actual cost to about 1.71 bits per weight. Compared with the standard 16 bits, this is a compression ratio of about 9.4x.
This compression approach runs through all matrix-computation-intensive components in the model: the embedding layers, attention projections, MLP projections, and the language model output head. Only a small number of normalization and scaling parameters remain at higher precision. PrismML emphasizes that the Bonsai series is a low-bit re-representation of existing open-source models, not pretraining from scratch.
Bonsai 2 27B Benchmark Results: What Is the Cost of Compression?
According to data released by PrismML, the performance of Bonsai 2 27B on 15 benchmarks (thinking mode) is as follows:
- Mathematics: 93.40 (original Qwen was 95.33)
- Code: 85.96 (original 88.74)
- Knowledge and Reasoning: 76.96 (original 83.15)
- Agents and Tool Calling: 74.01 (original 80.00)
- Instruction following: 71.77 (original 78.47)
- Vision: 65.19 (original 72.61)
Overall, the ternary version retains about 94.6% of the original model’s performance. By comparison, the binary version also offered by PrismML further reduces the size to 3.9 GB, but its performance retention rate drops to 89.5%.
Traditional low-bit quantization methods (such as Q4_K_XL and IQ2_XXS) exhibit selective collapse on certain benchmarks. For example, IQ2_XXS (2-bit) drops to 57.5 points on the AIME26 math test and to 56.4 points on LiveCodeBench, but still scores 88.93 on MMLU-Redux; short-form question-answering benchmarks mask its severe degradation on complex reasoning tasks. PrismML’s ternary compression approach shows relatively gradual attenuation in this regard.
Memory is the core bottleneck.
Running a model on a phone involves more than just storage space. iOS imposes a hard limit on the memory usage of a single app, around half of physical memory. On an iPhone with 12 GB of memory, a single app can actually use only about 6 GB.
The KV cache is another critical budget item: in the Bonsai 27B architecture, only 16/64 layers use full attention caching, requiring about 64 KiB per token under FP16. With a 262K-token context window, the KV cache requires about 17.2 GB; using a 4-bit KV cache can compress it to about 4.3 GB.
Throughput test results on different hardware platforms:
- M5 Max（Binary）Generation 66.4 tok/s, prefill 874 tok/s
- M5 Pro（Ternary）Generation 26.2 tok/s, prefill 393 tok/s
- iPhone 17 Pro Max（Binary）Generation 11.0 tok/s, prefill 111 tok/s
- H100 GPU（Binary）: Generation 104.8 tok/s, prefill 2,755 tok/s
The official website also advertises that Bonsai 2 delivers 8x higher throughput and 5x lower energy consumption than the original model.
This is where higher retention matters most: fewer derailments across multi-step tasks, fewer silent failures as the state evolves, and better consistency from one decision to the next.
Here is Ternary Bonsai 2 27B running an agentic coding workflow with Cline on an NVIDIA… pic.twitter.com/lqaE2bktQ1
— PrismML (@PrismML) September 17, 2026
It’s not just about compressing size; agent capability is the key breakthrough.
The biggest change in Bonsai 2 compared with the first generation is that the overall performance retention rate increased from 95% to 98.2%, but its advances in specific areas are even more critical. PrismML particularly emphasizes that the new model shows significant improvements in three areas: agentic coding, multimodal reasoning, and long-horizon tool use.
These three capabilities are exactly what the current AI Agent ecosystem needs most: writing and modifying code, understanding mixed image-and-text input, and maintaining context across multi-step tasks. The biggest weakness of past quantized models has often been these kinds of complex reasoning tasks; compressed models perform well on simple Q&A, but once multi-step logic is involved, they degrade severely. Bonsai 2’s improvements in this area give it what it takes to serve as the brain of an on-device Agent.
Computer use is another strong test.
The model has to repeatedly interpret state, choose the next action, and stay coherent across a long sequence of steps, exactly where small capability gaps become visible.
Here is Bonsai 2 27B running a computer-use workflow locally on the… pic.twitter.com/SlFwk6WZo7
— PrismML (@PrismML) September 17, 2026
In tool-calling benchmarks, the ternary version of Bonsai 2 scored 74.01 out of 100. Although it trails the original model’s 80.00, it is far above how conventional 2-bit quantization performs on similar tests. An independent evaluation by GitHub user MiaAI-Lab also corroborated this: the ternary version reached 95.2% and 88.4% of the original level on easy and medium-difficulty tool-use tasks, respectively. However, some users on X also reported that PrismML exaggerated Bonsai 2’s capabilities, and that its real-world performance was not as impressive as officially claimed.
98.2% of Qwen3.8 27B my ass.
I got hyped and gave it a real agent job right away. Build me an FPS in three.js, 6 hours on a 3090. My most standard and default prompt that I always use. It spent the first 32K tokens on a plan without writing a single file, then shipped a black… https://t.co/4ki588L4sg pic.twitter.com/g1PVgfXiww
— Alexey Fateev (@superalesha) September 18, 2026
Who is using it? Real-world use cases
PrismML has planned four main use cases for Bonsai 2:
- laptop-side agentThe ternary version can fully handle code repository analysis with a 262K token context on a laptop, making it suitable for developers who need to work in a local environment.
- On-device inferenceA one-bit version can fit into devices such as the iPhone; the whitepaper measured that on an iPhone, every 1% of battery consumed can generate 672 tokens.
- Privacy-Sensitive and Offline WorkflowsAll data stays on the device and never goes through the cloud.
- Single GPU serviceWith a 4-bit KV cache, 27B-class quality can be served on a graphics card with 24 GB of VRAM.
Downloads have surpassed 10 million, and the next step is the 10-billion-parameter scale.
PrismML’s first-generation Bonsai models (released in March this year) have accumulated over 11 million downloads, and with 2.6 million for its other smaller variants, total downloads across the entire family have surpassed 13.6 million. The first-generation Bonsai had a benchmark retention rate of 95%, while Bonsai 2 raised that figure to 98.2%, showing that compression technology continues to improve.
PrismML plans to release compressed models at the tens-of-billions-of-parameters level in the coming months. “The larger the model, the more room there is to preserve intelligence during compression. For larger models, it is easier to approach 100% performance retention.” PrismML’s technical roadmap is to build a complete product line of compressed models ranging from small to large, rather than just making one or two phone models that can run.
Bonsai 2 27B is already available. Hugging Face Now available, licensed under Apache 2.0, with support for llama.cpp (CUDA, Metal) and Apple’s MLX inference framework.
Source: KOCPC Chinese