Recently, an independent developer created a pure C language inference engine called Colibri (colibrì), successfully running GLM-5.2, an open-source MoE (Mixture of Experts) model with 744 billion parameters (744B), on a consumer laptop with only 25GB RAM. This project Hacker News received over 730 points and 180 discussions, being praised by the community as “the ultimate demonstration of hacker spirit.” Colibri’s core is only about 2,400 lines of C code, with zero dependencies – no Python needed, no GPU needed, no BLAS math library needed. The developer stated: “Speed is not the point, the journey to achieving this goal is. I just wanted to make it run, even if it’s slow doesn’t matter.”
Colibri: Running GLM-5.2 with 744 billion parameters on a 25GB RAM laptop
What is GLM-5.2? Why can it run on low memory?
GLM-5.2 is an open-source large language model developed by Chinese AI team Z.ai, and is also the most capable open-source model to date. It has 744 billion parameters and was trained on 28.5 trillion tokens, achieving top-tier performance across multiple benchmarks. If implemented using a traditional dense model architecture, simply loading the model into memory would require over 1.5TB of RAM, far beyond what consumer-grade hardware can support.
GLM-5.2 uses a Mixture of Experts (MoE) architecture, which is the key to its ability to operate in low-memory environments. The core concept of MoE models is that although there are 7.44 trillion parameters in total, only about 400 billion parameters (approximately 5%) are activated when processing each token (a unit of text). The remaining parameters exist in the form of “experts” and are only called upon when selected by the router.
Colibri’s developers leveraged this characteristic: during each token processing, only about 11GB of weights change (i.e., the selected routed experts). The remaining dense layers (attention mechanism, shared experts, embedding layer, etc., totaling approximately 17 billion parameters) can stay resident in memory. This enables Colibri to run on a machine with 25GB of RAM.
I managed to run GLM-5.2 (744B MoE) on a humble 25 GB RAM laptop — pure C, experts streamed from disk
byu/Just_Vugg_PolyMCP inLocalLLM
Colibri Technical Implementation
Colibri’s operational strategy is quite intuitive: dividing the model into two parts for processing. The dense portion (attention layers, shared experts, embedding layers, and approximately 17 billion parameters) is quantized to int4 and kept resident in RAM, occupying approximately 9.9GB. The remaining 21,504 routed experts (from 75 MoE layers × 256 experts, plus MTP prediction heads) are stored on disk, totaling approximately 370GB, with each expert taking approximately 19MB, loaded on-demand via streaming.
To boost efficiency, Colibri implements several optimization techniques: an LRU cache mechanism per layer keeps frequently used experts in memory; asynchronous expert readahead overlaps disk reads with matrix operations; and router-lookahead prefetch leverages 71.6% routing predictability between adjacent layers to preload experts that the next layer may need.
Colibri also implements GLM-5.2’s native MTP (Multi-Token Prediction) speculative decoding. The MTP head at layer 78 first drafts multiple candidate tokens, which are then batch-validated by the main model. With int8-quantized MTP heads, draft acceptance rates reach 39% to 59%, generating 2.2 to 2.8 tokens per forward pass. However, the developers acknowledged that under cold cache conditions, speculative decoding actually becomes slower due to the need to load additional experts.
Additionally, Colibri implements MLA attention mechanism (with compressed KV cache, requiring only 576 floating-point numbers per token instead of 32,768—a 57x reduction), DSA sparse attention (selecting only the top-2048 causal keys per layer), and int8/int4 integer matrix operation cores (accelerated with AVX2 instruction set). The KV cache also supports persistence across restarts, allowing conversations to seamlessly continue after the engine restarts.
Actual Performance Data
The developer’s measurements on their own machine (WSL2, 12 cores, 25GB RAM, NVMe via VHDX) are as follows: model loading time approximately 30 seconds, resident memory 9.9GB, peak RSS during chat around 20GB (auto ceiling). During cold start, each token requires reading approximately 11GB of disk data (75 layers × 8 experts), and under the VHDX random read limitation of approximately 1GB/s, cold inference speed is approximately 0.05 to 0.1 tokens per second.
After the warm-up period, with MTP speculative decoding and thermal expert pinning, practical response latency decreases significantly. Community reports indicate that with faster NVMe and more RAM configurations, speeds exceeding 1 token per second can be achieved. The developer emphasized: “I don’t have better hardware, so I can’t test on more powerful machines.”
Colibri GitHub Project URL
Limitations and Challenges of Colibri
While technically impressive, Colibri is still far from practical use. The first issue is speed: during cold start, it only generates 0.05 to 0.1 tokens per second, meaning a 100-token response would take 15 to 30 minutes. Even under optimal conditions with warm cache and NVMe, it only reaches about 1 token per second—far below the experience of cloud APIs or GPU inference.
Storage space is another consideration: the int4 quantized model still requires approximately 370GB of disk space, compared to 756GB for the original FP8 model. While the conversion tool can download and convert one shard at a time, eliminating the need to have the full 756GB available at once, a 370GB model file still represents a significant chunk of storage for most consumer SSDs.
Third is the SSD lifespan and cooling issues. The developers admit in the README that cold starts require approximately 11GB of random reads per token. Although reads don’t wear out SSDs the way writes do, prolonged full-load reading can cause lower-end SSDs to heat up. Additionally, if the system RAM is insufficient and triggers swap writes, it will affect the SSD’s lifespan.
Fourth are the practical limitations of MTP speculative decoding. MTP heads must use int8 quantization (with int4, draft acceptance rate is only 0% to 4%), and under cold cache conditions, each draft verification requires loading more experts (from approximately 660 to approximately 1,100), which may actually result in a net time loss. The developer provides a DRAFT=0 option to disable this feature.
Fifth is the lack of GPU acceleration. Although Colibri has experimental CUDA layer support, the core engine relies entirely on CPU computation and cannot leverage the massive parallel computing power of GPUs. For users who have GPUs, mature frameworks like llama.cpp may be a more practical choice.
Conclusion
Colibri’s value lies in proving how far the architectural properties of MoE models can be pushed, not its speed or practicality. A Hacker News user accurately summarized that this project demonstrates MoE inference can be achieved on consumer hardware. In an interview, the developer mentioned that his inspiration came from antirez’s ds4 project (the author of Redis), which also explores running GLM-5.2 through disk streaming. For scenarios requiring local execution of large models, more practical choices might be using a Mac Studio with 256GB or more, a multi-GPU workstation, or simply relying on cloud APIs.
Source: KOCPC Chinese