The most widely used Chinese Alibaba in the open source community Qwen The Qwen model team, following its strongest 2.4T parameter-scale Qwen3.8 After Max’s model weights were opened, on the evening of August 14th Taiwan time, the official team continued releasing the most widely used native multimodal 27B parameter dense model. Qwen3.8-27BAlmost simultaneously, Unsloth AI, known for its quantization and local inference tools, also announced a 4-bit quantized version of Qwen3.8-27B that can run locally, requiring only a consumer-grade GPU with 17GB of memory.
Qwen3.8-27B: A Native Multimodal Dense Model
Qwen3.8-27B According toOfficial explanationIt is a native multimodal dense model that integrates vision and language capabilities, with native support for a context length of 262,144 tokens (256K), extendable to 1 million tokens via YaRN. In terms of positioning, the Qwen team links the Qwen3.8 family with the previous generation Qwen3.5: it continues the hybrid architecture approach of Gated Delta Networks and sparse MoE, but with increased investment in training data and reinforcement learning scale, and implements vision-language fusion as early fusion training, making the model more aligned with real-world usage in scenarios such as coding, agentic workflows, and document understanding. In terms of capabilities, the officially announced results significantly surpass those of the much larger previous-generation Qwen 3.7-Plus, with some results even exceeding Opus 4.6 MAX—quite impressive.
The Agent’s performance is exceptional, nearly leading across the board over Opus 4.6 MAX:
The Qwen3.8 family is licensed under Apache 2.0, meaning commercial use, redistribution, and integration into products require no additional applications. For teams looking to embed the model into internal toolchains, the licensing risk is significantly lower than with custom terms.
Hardware Requirements: Tiers from 11GB to 56GB
The quantized versions of Qwen3.8-27B are differentiated by total memory:
- 11 to 13GBThe minimum runnable extreme quantization range, suitable for lightweight testing.
- 13 to 16GBEntry-level usable range, with completion and speed beginning to stabilize.
- 17-19GBThe sweet spot of 4-bit quantization, where the RTX 5080, RTX 4090, or 24GB Mac all fall within this range.
- 24GB、31GB、56GBHigher precision quantization comes with higher memory requirements, and offers better accuracy and long-context stability.
According to the file listing on Hugging Face, Unsloth provides over 20 GGUF variants at once, covering Q4_K_M, Q5_K_M, Q6_K, Q8_0, IQ4_XS, UD-Q4_K_XL, UD-Q8_K_XL, and more, and additionally offers the mmproj vision projection file, meaning multimodal capabilities can be enabled directly locally. BF16 full precision is provided in two shards, totaling about 16.3GB, while the quantized Q4_0 and Q4_K_M are even smaller in size. In other words, 17GB is not a single magic number, but a result jointly determined by quantization options and cache configuration.
Qwen3.8-27B Hugging Face
As for the giant model Qwen3.8-2.4T-A95B in the family, the requirements are completely different. The guide indicates that even with the new 1-bit quantization, it still requires five tiers: 397GB, 508GB, 657GB, 2.6TB, and 4.9TB; full precision requires 4.9TB of storage space (likely no one can deploy it locally).
How Unsloth’s 17GB came about: Dynamic GGUF V3.0 and NVFP4
And for general use, the Unsloth quantized version Qwen3.8’s GGUF Adopts the Unsloth Dynamic V3.0 Preview, maintaining leading accuracy even after quantization, and adds support for agent tools.
- Dynamic GGUF V3.0Unsloth’s quantization format tuned for Qwen3.8 focuses on preserving more detail at lower bit widths. The official guide also mentions that the quantized version of Qwen3.8-27B supports the Developer role and improved tool-call parsing, enabling more stable handling of nested objects and increasing tool call success rates in agent frameworks.
- NVFP4Offered alongside GGUF, a 4-bit floating-point quantization for the NVIDIA GPU ecosystem. Unsloth releases both formats simultaneously, letting users choose based on the inference backend: llama.cpp uses GGUF, while TensorRT-LLM or vLLM-related stacks use NVFP4.
- 1-bit familyTo make it possible to discuss a giant 2.4-trillion-parameter model, Unsloth extended llama.cpp’s IQ1_S architecture, compressing from 1.5625 bits per weight down to 1.1875 bits. The official table lists new types such as UD-IQ1_XXS and UD-IQ1_XXXS, among which the 397GB Q1_0 version is designated as the recommended quantization for the 2.4T model, with a size 91% smaller than the original 4.9TB.
Unsloth founder Daniel Han verified the 17GB estimate earlier on the LocalLLaMA community. The calculation is based on model weights at roughly 17GB, plus about 2GB of KV cache, assuming 4-bit and an 8192 context. This aligns with the 17–19GB range stated in the documentation, with the difference mainly coming from context length and cache settings.
Unsloth quantized version Qwen3.8 27b
How to get it running: Two paths, Unsloth Desktop and llama.cpp
The official has consolidated the local execution paths into two, covering users with a graphical interface and pure command-line users:
- Unsloth DesktopAn open-source local AI interface supporting macOS, Windows, and Linux. It integrates MLX and llama.cpp, allowing you to search, download, and run GGUF and safetensors models from a graphical interface, while automatically managing RAM and VRAM offloading and detecting multi-GPU configurations. Installation options include a downloadable installer and a one-line command (for macOS and Linux).
curl -fsSL https://unsloth.ai/install.sh | shWindows, on the other hand, executes the corresponding scripts via PowerShell. This is currently the shortest path—simply search for Qwen3.8 in the interface to select a quantized version for download and inference. - llama.cppFor users who need to compile and integrate into their own backend. The guide provides compilation instructions, using CMake in a Linux environment, and distinguishes between enabling or disabling CUDA. If you only need standard quantizations like IQ1_S, you can compile directly with the regular llama.cpp. If you need Unsloth’s extended new types from 1.1875 to 1.5625 bits, you’ll need to switch to
iq1-narrowFork and build tools such as llama-cli, llama-server, etc. - Hugging Face direct useQwen official weights are located at Qwen/Qwen3.8-27B, Unsloth quantized versions are at unsloth/Qwen3.8-27B-GGUF, and Alibaba Cloud users can also get them via the ModelScope mirror.
The document also mentions performance expectations: on data center cards like the B200, it can achieve generation speeds of about 20 tokens per second and throughput exceeding 120 tokens per second. The rule of thumb is that it runs smoothest when the combined RAM and VRAM roughly equals the quantized file size; otherwise, disk paging slows things down. This aligns with what local users care about: VRAM isn’t the only metric—whether unified memory and system memory can fill the gap also affects the experience.
I personally have also tested it in my RTX-5090 Laptop 24GB Gaming LaptopMoreover, with MTP it can achieve 5X t/s performance, and if you don’t attach a vision model, you can run it smoothly at 128K context. Hermes handles complex tasks well—it’s currently one of the best local model options out there. If you have data sensitivity concerns, you can test it out yourself:
Source: KOCPC Chinese