China AI Startup Moonshot AI(Dark Side of the Moon) was officially open sourced on July 27 Kimi K3 Model weight is also the first in history to have the first-tier closed-source AI capabilities of OpenAI and Anthropic.Open source model. Therefore, both capable large-scale enterprises and individual AI players want to test whether this MoE model with 2.8 megabytes of parameters can “run it by themselves” on the local side. The test has been successful, but the cost may be beyond ordinary people’s imagination.
X Well-known AI player @totheagi (Ning) took the lead in demonstrating the results of using 80 RTX 5090s to execute K3: a single stream of 20 tok/s, without tuning on the first day. He emphasized that this system does not use HBM high-bandwidth memory at all, and only relies on GDDR7 game graphics cards, ordinary Ethernet and official MXFP4 weights to run the most powerful open source model on the planet.
we got the full Kimi K3, 2.8T params, running on 80x RTX 5090s.
20 tok/s single stream, day one, untuned. Last week we took GLM-5.2 from 30 to 110 tok/s on this same fleet. This number will climb.
A first for open weights: frontier intelligence served with zero HBM, the… https://t.co/PPIuTCRsyT pic.twitter.com/ZqNEozTEXV
— Ning (@totheagi) July 27, 2026
However, 80 RTX 5090 is obviously not “general hardware” for most people. What level of equipment is needed to deploy Kimi K3 yourself? How much hardware is needed just for the step of “loading the weights into memory”? How much video memory does the KV cache and context consume? Next, we will sort out the “prices” that you may need to prepare for various subordinate plans.
How much is the weight? Three formats, three numbers
The 2.8 MB parameters of Kimi K3 will vary due to different storage formats.completely differentFile size:
- MXFP4 (native publishing format): Approximately 1.4 TB. The attention layer, shared expert, dense layer and visual tower retain higher accuracy, and only the MoE routing expert is compressed with 4-bit. This is the smallest official full quality version
- MXFP8: Approximately 2.8 TB. All weights are stored in 8-bit, and the file size is exactly equal to the number of parameters in megabytes. This is also the source of the “2.8 TB version” mentioned in many community discussions.
- BF16/FP16 full precision: Approximately 5.6 TB. The theoretical size if all 2.8T parameters are stored in 16-bit floating point
The actual official download weight on HuggingFace is 96 safetensors archives, totaling 1,561 GB (~1.56 TB). This is a hybrid format: attention and dense layers retain BF16 accuracy, and MoE expert layers are compressed with MXFP4. Someone in the HuggingFace discussion forum actually tested the dtype of the safetensors file and confirmed it to be BF16, indicating that the official download version is not pure MXFP4.
How much video memory is needed just to load the weights?
Taking the official deployment of vLLM as an example, the model weights alone require approximately 1,561 GB of VRAM. But this is not enough, additional space is needed for actual execution:
- Model weight：~1,561 GB
- KV cache: Varies depending on the context length and number of concurrencies. K3’s KDA architecture allows KV to grow approximately 29 times slower than traditional full attention models. The KV cache of a single sequence and millions of token contexts is about 14.5 GB (FP8), but it will accumulate quickly in concurrent scenarios.
- Boot buffering and CUDA graphs: ~15 GB per GPU
- Communication buffer (multi-node): Varies according to network topology
vLLM official estimateofMinimum VRAM requirement is 1,680 GB, This is just the bottom line of “being able to load weights and run”, and does not take into account high concurrency or long context scenarios.
Memory cost of KV cache and context
K3’s KDA (Kimi Delta Attention) hybrid attention architecture is a memory-friendly design: only 24 of the 93 layers have KV cache (about 13.5 KB/token@FP8 after MLA compression), and the remaining 69 layers use fixed-size loop states (about 127 MB/sequence, does not grow with the context length).
Estimating KV cache requirements with different context lengths (single sequence, FP8):
- 8K context: almost negligible (< 0.2 GB)
- 32K context: Approximately 0.5 GB
- 128K context: Approximately 2 GB
- 1M context: Approximately 14.5 GB (MLA layer) + 127 MB (KDA fixed state) ≈ 14.6 GB
But concurrency is the main source of stress. For each additional concurrent sequence, the KDA layer takes up an additional 127 MB of fixed state. 100 concurrent users = 12.7 GB of pure KDA state, plus the MLA layer’s KV cache grows with the context. The official multi-node base test uses –max-num-seqs 512 as the measured concurrency upper limit.
Complete memory calculations for different GPU hardware configurations
Configuration A: 8× B300 (192 GB/photo) = 1,536 GB
This is the minimum single-node configuration officially recommended by vLLM. 192 GB each, 8 total 1,536 GB. However, vLLM requires a minimum of 1,680 GB, equal to144 GB left, 8-card single node is strictly not enough. Some features need to be turned off (such as skipping the visual encoder and shrinking the KV cache pool) to barely perform.
Configuration B: 16× H200 (141 GB/photo) = 2,256 GB
Two-node configuration, 141 GB HBM3e each, 16 sheets total 2,256 GB. After deducting the 1,561 GB weight, 695 GB remains available for KV cache and buffering. Calculated based on millions of token contexts and approximately 14.6 GB per sequence, approximately 40-45 long context sequences can be maintained at the same time. This is currently the most common production-level deployment scenario.
Configuration C: 16× B200 (192 GB/photo) = 3,072 GB
Two-node Blackwell configuration, 3,072 GB total. After deducting the weight, the remaining 1,511 GB can support higher concurrency and longer contexts. The Blackwell architecture supports FP4 computing acceleration, but Hopper does not.
Configuration D: 80× RTX 5090 (32 GB/photo) = 2,560 GB
This is @totheagi’s plan at the beginning. After deducting the weight, 999 GB is left, which is enough space. The RTX 5090 uses GDDR7 instead of HBM, and the bandwidth of a single memory is about 1.8 TB/s (vs. 4.8 TB/s of the H200), which is the main bottleneck at the 20 tok/s speed.
Complete cost comparison table
Here’s the full VRAM requirement for “load weights + KV cache + buffering” versus hardware cost:
- 8× B300（1,536 GB）: Cloud rental is US$42-78/hour, monthly rental is US$30,000-56,000 (approximately NT$970,000-1.81 million). VRAM is tight and needs to be trimmed to perform
- 16× H200（2,256 GB）: Cloud rental is US$42-158/hour, monthly rental is US$30,000-73,000 (approximately NT$970,000-2.36 million). Production-grade configuration, with room to handle concurrency
- 16× B200（3,072 GB）: The cloud rental price is similar to or slightly higher than H200. Blackwell accelerates FP4 inference
- 80× RTX 5090（2,560 GB）: The hardware cost is about US$200,000-250,000 (approximately NT$6.48-8.1 million), and the monthly electricity bill is more than US$10,000 (approximately NT$320,000). No HBM required but speed limited by GDDR7 bandwidth
- Self-purchased H100 series (from 18 photos): Hardware cost US$450,000-720,000 (approximately NT$14.6-23.3 million), plus cabinet, cooling, electricity, and space
Hybrid offloading solution (consumer-grade GPU + large-capacity RAM): A 24 GB graphics card plus 768 GB server RAM, using IQ1_S extreme quantization (1.63 bpw), can achieve ~5-10 tok/s. The multi-channel memory architecture of the EPYC or Xeon platform is required, and the dual-channel DDR5 of ordinary desktop computers is completely unaffordable.
Conclusion: Using API directly is the best choice for normal people
For users whose average daily output is less than 1 million tokens, directly using the API may be the most reasonable choice. Moonshot official API pricing is $3 per million tokens for input ($0.30 for cache hits) and $15 per million tokens for output. China’s Alibaba Cloud Bailian platform has also put kimi-k3 on the shelves. The input price is 20 yuan per million tokens (cache hit is 2 yuan), and the output price is 100 yuan per million tokens (almost the same as the official pricing).
In comparison, the monthly rental fee for 16 H200s ranges from US$30,000 to US$73,000, which is approximately equivalent to the monthly API fee of 20 million to 50 million output tokens. If your usage is below this threshold, it is not economically efficient to rent or buy the hardware and build it yourself. In addition, self-built solutions also need to bear maintenance and operation risks: hardware failures, driver compatibility, vLLM version iterations (the official Docker image was updated three times within 24 hours), and continuous optimization of the model itself. The above hidden costs are often ignored during evaluation.
The open source of Kimi K3 has indeed lowered the threshold for obtaining top AI capabilities, but there is still a huge gap between “open source” and “general hardware that can run it.” Loading the weights alone requires 1.56 TB of VRAM or RAM, plus the KV cache and inference buffer, and the minimum threshold for vLLM is 1,680 GB. @totheagi used 80 RTX 5090s to prove that K3 can be executed without relying on HBM. However, the total cost of this configuration is still more than NT$5 million, plus hundreds of thousands of monthly electricity bills and site costs.
For most developers and small teams, hybrid offloading solutions or waiting for community quantitative solutions to mature are more pragmatic paths. And if you just want to experience the capabilities of K3, the cost-effectiveness of using the API directly is much higher than building your own hardware.
The figures in this article were calculated using publicly available information on the Internet. Please correct me if there are any errors.
Source: KOCPC Chinese