Recently, Tsinghua University in China released a set of open source framework GitHub projects called KTransformers, which enables the replacement of a $400,000 H100 cabinet with a 24GB graphics card to execute complete large-scale MoE models such as DeepSeek-V3 and R1. KTransformers has sparked heated discussions in the AI community, giving ordinary people without expensive hardware the opportunity to run very large-scale AI models locally.
Core idea: Run MoE as a “sparse model”
KTransformers was jointly released by Tsinghua University’s MADSYS Laboratory, Approaching.AI and 9#AISoft. It is licensed under Apache 2.0 and has been published at SOSP 2025, the top conference in the system field. Its core idea is based on the essential characteristic of a MoE (Mixture-of-Experts) model: sparseness.
Taking DeepSeek-R1 as an example, the total parameter volume is 671B, but when each token passes through the router, only a small group of experts will be activated, and only a few of hundreds of experts will be triggered. KTransformers takes advantage of this asymmetry to the extreme: keeping the hot path components (DeepSeek’s MLA attention layer and KV cache) on the GPU, and putting the MoE experts, which account for most of the parameters but have a very low trigger frequency for each token, into the system memory (RAM).
This means that the reasoning limitation that was originally regarded as “VRAM is a high wall” has been re-deconstructed into a multi-level memory hierarchy problem: GPU VRAM is the cache, system RAM is the main layer, and NVMe can even extend further down. A 24GB RTX 4090, plus about 512GB of system memory, would hold the full 671B model.
A similar approach is used in another open source project Colibri that we reported before, but Colibri puts the unactivated expert model in the SSD, resulting in very slow speed (but it can run).
The key to moving experts to the CPU: extremely fast CPU kernel
Moving experts from VRAM to RAM comes with a price: having to rely on the CPU to compute them. If the computation on the CPU side is too slow, the entire advantage is lost. The technical focus of KTransformers is a very fast CPU kernel that calls the Intel AMX and AVX-512 instruction sets for INT4 and INT8 quantized versions.
According to the project benchmark data, the AMX version of the MoE kernel is approximately 27.8 times faster than llama.cpp in the test of two 32-core CPUs (llama.cpp stopped at approximately 10.31 tokens/s in this test). Around this core, KTransformers has also done several things to make the whole thing practical: NUMA-aware memory management for multi-socket CPU systems; Marlin quantified GPU kernel plus FP8 weight, allowing 24GB VRAM to run about 139K token context; three-layer prefix cache (GPU, CPU, disk) to reuse completed calculations across memory layers.
In terms of actual performance, according to indicative figures from project documents and community testing, 671B prefill can reach approximately 286 tokens/s under optimal configuration; the output (generation) speed is slow and highly dependent on hardware. With a server configuration of 8 L20 GPUs and Xeon, the project reported approximately 227 tokens/s aggregate throughput and 87 tokens/s output speed (8 parallel requests).
More than DeepSeek: Model and Hardware Coverage
Starting with v0.6.3, released in June 2026, KTransformers’ list of support for open source MoE models reads like a list of open weight frontiers: DeepSeek-V3/R1, Kimi K2 and K2.5, GLM-5 and GLM-5.2, Qwen3-MoE, Qwen3-Next, and MiniMax series. The hardware is no longer limited to NVIDIA, but already covers AMD ROCm, Intel Arc and Huawei Ascend NPU.
In addition to reasoning, the project also moved fine-tuning to ordinary hardware. By integrating with LLaMA-Factory, a DeepSeek-V3 SFT running on 4 RTX 4090s and about 80GB of total memory can reach about 3.7 iterations/sec. The project reports that this is 6 to 12 times faster than similar solutions.
KTransformers project website
Comparison of similar solutions
KTransformers is not the only example of this kind of gameplay. The entire open source community is developing different entry points around the feature of “frontier MoE is large but sparse”. llama.cpp allows users to choose which tensors to keep in the CPU and which to put on the GPU through the –override-tensor parameter. It is the first tool to mainstream selective offloading; ik_llama.cpp is a branch of llama.cpp, focusing on the CPU kernel for quantization and MoE. PowerInfer separates “hot” and “cold” neurons on the GPU and CPU, while Fiddler specializes in optimizing the expert calculation path on the CPU.
The common logic is the same: use sparsity as leverage. Some rely on RAM (KTransformers), some rely on disk (colibri streams experts from NVMe), and some rely on hot and cold distribution (PowerInfer). The practical results are consistent: moving cutting-edge models from data centers to servers under desks is no longer a theory, but an achievable option.
Source: KOCPC Chinese