People who run local models probably all share a misconception: “the less quantized a model is, the smarter it is.” The common belief is that the original (BF16) version is definitely smarter than a quantized version, but recently Piotr Migdał, a well-known figure in the quantization community, published a large-scale empirical test on the Quesma blog: taking Qwen3.6-27B’s I tested all 24 GGUF quantized versions.The conclusion drawn from file sizes ranging from 9.6GB (UD-IQ2_XXS) to 54.7GB (the original BF16 version), running for 37 hours on a home MacBook Pro M5 Max to generate 3.9 million tokens, and then spending another $1,430 (about NT$46,000) on cloud GPUs may surprise you.
First, understand the four major quant schools.
Before testing begins, the article first explains several common quantization formats currently in use. The most common everyday “K series” is llama.cpp’s default; the filename suffixes _S/_M/_L/_XL indicate how many high-precision tensors are retained (the degree of conservativeness). Q4_K_M’s actual precision is roughly equivalent to 4.6 bits, and many tensors inside it are 6 or even 8 bits, so “it cannot be understood as all tensors being 4 bits.” The “IQ series” uses an importance matrix to spend precision where it matters most, and at the same size it is usually smarter than the K series; the “old antiques” Q4_0 and Q4_1 only round; most highly regarded is the Unsloth team’s “UD” (Unsloth Dynamic) dynamic quantization, which automatically identifies the tensors most sensitive to compression and gives them more bits, with precision closest to the original. In the floating-point camp, FP8 is almost equivalent to the original, while NVFP4 is NVIDIA’s own 4-bit floating-point scheme. If you don’t know which quantization format to choose, the order suggested in the article is simple:If UD is available, choose UD; if not, look at IQ, then Q4_K_M. Keep in mind that file sizes vary greatly between quantization formats.。
KL Divergence: A Metric 100 Times More Reliable Than Bit Count
Next, we’ll discuss what some numerical values you may have heard of but don’t understand the meaning of represent, such as perplexity (lower is better). The BF16 original is 6.40, and Q4_K_M is 6.45, “basically can be considered the same model”; all quantizations above Q3_KM are clustered between 6.39 and 6.5. Only further down do they start to degrade: Q3_K_S: 6.7, Q2: 7.0, IQ2_XXS: 8.19.
But the author offers more practical selection criteria:KL divergence(the degree to which the quantized version’s output distribution deviates from the original). The boundary is surprisingly clean: below 0.05, it stays close to the full version and can be used with confidence; above 0.08, there is a real gap. Coincidentally, the one that lands exactly at 0.05 is Q3_K_M again, while Q3_K_S jumps all the way to 0.08.
Two other metrics are also worth noting: the Top-1 agreement rate at temperature 0—Q8_0 is as high as 99.3%, Q4_K_M is 95.1% (about a 5% difference from the original), and Q2 drops to 78.7%. And with temperature set to 1, asking the same model twice, even BF16 and “itself” have only a 48.5% chance of choosing the same token. This means:Different output doesn’t mean it’s gotten dumber—that’s called randomness.The worst deal is FP8: its KL divergence of 0.017 is almost the same as Q4_K_M’s, yet the file is 31GB versus 17.1GB—nearly double the size for the same similarity.
Benchmark: The cliff didn’t arrive—noise did.
The results from two authoritative test sets are even more striking: on Terminal-Bench 2.1 (89 real terminal tasks that test AI programmers’ hands-on capabilities), Q4_K_M achieved a pass rate of 40.4%, evenEven higher than the original version.Q5_K_M was only 29.2%, even trailing 2-bit. But statistics is unforgiving: the p-values for all these differences were non-significant, “I expected to see a performance cliff, but all I saw was noise.” In other words, on Agent tasks, nothing from Q2 to BF16 forms a bottleneck.
The 120-question competition math AIME is the real watershed. The experimental setup follows the paper Quantization Hurts Reasoning? with temperature 0.6 (temperature can be viewed as creativity) and a 32K token limit. The original BF16 solved 70.8% (85 questions), and UD-Q4_K_XL was exactly the same at 70.8%; Q3_K_M even reached 73.3%, three percentage points higher (not significant). But 2-bit really fell apart: IQ2_XXS scored only 32.5% (39 questions, p < 0.001, statistically significant), and the other Q2 variants were only between 52% and 57%. The author also casually drew an analogy: this 27B BF16 score is already close to OpenAI o1’s 74.4% in September 2024.
For those impression-based claims like “I can clearly feel that Q8 is smarter than Q4,” the test results are straightforward: on authoritative benchmark sets, the two models score exactly the same, and “your feeling may be wrong.”
Pelican-drawing blind test: the rankings are completely random.
The most entertaining part is the SVG blind test: each quantized model is asked to draw five “pelicans riding bicycles,” and Gemini acts as the judge, scoring them in head-to-head matchups. This is the community-recognized classic SVG benchmark. The results completely scrambled the rankings from full precision to 4-bit: Q6_K scored +2.54, higher than BF16’s +1.76; UD-Q4_K_XL scored +0.48; the only one clearly falling behind was IQ2_XXS (negative score). Even NVIDIA’s NVFP4 (official and Unsloth versions), which NVIDIA claims is close to the original, produced worse drawings than the roughly 17GB Q4_K_M and Q4_K_XL.
The second challenge, “five-gear belt drive,” is even more absurd: the full-power version burned through its token budget without finishing the drawing; Q8_0 produced an empty output outright (it spent more than six times the tokens of the 4-bit version—an overthinking problem), while Q4_K_M finished the drawing and was the most stable in terms of completion.
The speed myth: quantization buys you memory, not speed.
“The lower the quantization, the faster it runs?” A real-world test proves that wrong. On M5 Max, the fastest was actually the oldest Q4_0 (34.7 tokens/sec), and even the slowest, BF16, hit 24.9. All 24 versions were squeezed between 25 and 35: a 3.4x difference in size, but only a 1.4x difference in speed. The reason is physical: what is stored on disk is 2-bit or 4-bit, but during computation every matrix multiplication has to be unpacked into 16-bit floating point; the saved size does not amount to much relative to the compute. The same quantization on different hardware: M5 Max 30 tokens/sec, NVIDIA L40S 94, H100 112.If you want more speed, spend money on a GPU; if you want to save memory, then look at quantization.。
Homework copying list
- 4-bit and above has no perceptible difference from the full-power version, and Q3_K_M can be viewed the same way.
- If you don’t want to think about it, choose Q4_K_M or UD-Q4_K_XL (about 17GB, 3 times smaller than the original).
- For limited memory, choose Q3_K_M; avoid Q3_K_S—the loss is significant.
- Check the KL divergence before downloading; if it’s below 0.05, just buy it blind.
- 2-bit really does make it dumber, so don’t force it (except for non-traditional architecture 2-bit approaches like Bonsai).
- Low quantization does not mean faster; the speed bottleneck lies in memory bandwidth and compute power.
[Analysis] You think you’re getting dumber—90% of it is an illusion.
The most subversive thing about this test is that it separates “feeling” from “measurement.” Five sets of evidence—perplexity, KL divergence, Top-1, blind testing, and benchmarks—interlock and point to the same conclusion: at the 27B level, quantization above 4-bit already feels like lossless compression, and the red line falls between the middle of 3-bit (Q3_K_S) and 2-bit. For people running local models, this directly rewrites the file-selection logic: instead of trusting the “high-precision peace of mind” of Q8 and FP8, take a look at KL divergence and use the memory you save to load a larger model or a longer context.
A caveat is also warranted: this conclusion is based on a single architecture, Qwen3.6-27B, and the authors themselves note that different models may respond differently; AIME has only 120 questions and Terminal-Bench has only 89 tasks, so the error bars are inherently wide. But at least the claim that “quantization makes models dumb” does not hold up in this data—and in some cases, the opposite is true. (Editor’s note: But based on my personal testing experience, models at different quantization levels do differ in capability, and with post-training they can indeed be better than the original; however, Q4_K_M is indeed a safer choice.)
Source:Do Qwen3.6 27B quantizations break the pelican?
Source: KOCPC Chinese