Barely three weeks after launch, the DIY inference engine Strata has pulled in 24,000 stars, and now it has dropped another major update. Author Niko Veit announced v0.1.44 on X: first-time Apple Silicon support, another overall speedup of 10% to 20%, and the addition of the Qwen3.6-35B-A3B model plus a batch of quantization packages, while also fixing about 20 bugs and merging 15 community PRs. The highlight of this release is the Mac version many people had been asking for, and AMD and Intel decoding speeds are faster too.
Strata Origin: If it doesn’t run well, write your own.
The reason Strata came into being is actually quite simple: the author Niko Veit When he first used a 12GB RTX 5070 to run the 176B Qwen3.8-Flash-Next, the VRAM was too small, so it ran at only about 15 tokens/s. He decided to write an entirely new inference engine from scratch. The resulting Strata is built in C++, MIT-licensed, and fully open source. The point of choosing C++ was to squeeze every last drop out of the hardware, and the goal was clear: enable gaming cards with 12 to 24GB of VRAM to run server-grade large models. The project was created on September 24 and now has 24,502 stars and 2,485 forks.
Strata – GitHub
How It Did It: Turning an Entire PC into a Graphics Card
How Strata works is worth unpacking; this is exactly where it sets itself apart from other architectures. Qwen3.8-Flash-Next is a mixture-of-experts (MoE) architecture (Strata was originally an inference framework dedicated to Qwen 3.8), and the entire model consists of 24,576 small experts. Generating each token requires using only about 10 of them, so expert utilization is inherently low. Strata seizes on this characteristic to do tiered scheduling; the official analogy is a kitchen: frequently used items stay on the countertop, while the rest go into the pantry. Specifically, it has three tiers: the GPU keeps the several thousand most frequently used experts, so popular ones are available on demand; system RAM holds all experts as the home base, while the CPU also shares the remaining computation; and the SSD holds a large lookup table recording who is who and where they are. When generating answers, hot experts compute directly on the GPU, and only when a colder expert is encountered is it moved on the fly from RAM or SSD. PCIe bandwidth is spent where it matters most. The whole machine loads about 35 to 55GB of memory and locks a portion of it for the GPU.
The second killer feature is “guess then verify”: a small assistant model first guesses the next few characters, and the large model verifies them all in one go. If guessed correctly, keep them; if wrong, replace them. Answer quality stays the same, and output speed is 1.6 to 1.8 times faster. For long documents, it can take in up to 8,192 tokens in one go, which is how it achieves a reading speed of over 1,000 tokens per second. Combined with quantization that compresses weights to 2 to 3 bits, only then can a 176B model fit into 12GB of VRAM. The UD big package download size is as high as 94GB, and when RAM is under 80GB, it can still fetch data from an SSD while answering, with the best experience on NVMe.
Compared with the two traditional approaches, cramming the whole thing into VRAM requires a multi-GPU setup starting at 48GB, while dumping the whole thing into RAM and moving it token by token is too slow to be usable; Strata’s “Hot experts on GPU, full data in RAM, lookup tables on SSD.The three-layer architecture is equivalent to assembling near-server throughput from the parts of an ordinary computer. They’ve even written an official research paper: the compression technology comes from ISTA-DASLab, UkisAI, and Unsloth, while the engine reuses some components from llama.cpp/ggml. Just look at the architecture and you can tell it’s serious engineering.
What can it do?
Strata’s biggest contribution to the AI world is successfully cramming the 176B-parameter Qwen3.8-Flash-Next into an ordinary gaming PC: it can handle chatting, writing code, and reading images with ease, and it can run on localhost through OpenAI- and Anthropic-compatible APIs, letting your existing tools and coding agents connect directly, with data never leaving your machine. The company repeatedly emphasizes “Nothing leaves your PC,” and privacy is one of its selling points.
Official real-world tests on two common cards: RTX 5070 (12GB) running Q2_0 generates answers at 94 tokens/s and reads 32K prompts at 2,650 tokens/s; IQ2_XS at 79 tokens/s, IQ3_XXS at 62 tokens/s, and the code-specialized Coder also at 55 tokens/s; AMD RX 9070 XT with Q2_0 starts at 60 tokens/s. The more VRAM, the faster; RTX 3090 is estimated to reach 100 to 140 tokens/s. Behind these codenames is quantization prowess: compressing model weights to 2 to 3 bit is the only way a 125B giant fits into 12GB of memory. The tradeoff is slightly reduced precision, in exchange for the reality that gaming cards can run it. The requirements are spelled out plainly: specified NVIDIA or AMD cards with 12GB+ VRAM, 32GB+ RAM (64GB recommended), and about 80GB of disk space; 64GB of memory can run all sizes, and an SSD is recommended for storage, as the first startup is much faster.
Its significance for the PC ecosystem is this: previously, 176B-class models were residents of server rooms, and ordinary gamers had only two ways to get their hands on them: rent cloud GPUs and wait for the bill, or save up for a workstation. Strata cuts the barrier down to “the gaming card you already have.” A 12GB RTX 5070 can run it, a 16GB RX 9070 XT can run it, and even the RX 6800/6900 from two years ago are still on the support list, letting mid- and low-end Windows PCs get their hands on this class of model for the first time. Even the AMD camp, which previously did not get much attention, gets equally solid treatment, spanning from the RX 7700 XT, 7800 XT, and 7900 series all the way to the latest RX 9060 XT and 9070 series. This time, v0.1.44 pushes multi-GPU configurations and decoding speed further, and adds Intel A-series support, so almost all consumer GPUs from major brands can now use Strata to run Qwen3.8-Flash-Next.
What was updated in v0.1.44?
This time, the focus is divided into four parts:
New update is here! – Apple Silicon is now supported – NVFP4 Support, UD-Q4, UD-Q5, UD-Q6, UD-Q8 – Qwen3.6-35B-A3B is also now available – Another 10%-20% boost and more: https://t.co/Ja3zgxkTgP https://t.co/IeJyOZNHXy
— Niko Veit (@coldniko) October 13, 2026
First, Mac support is finally here: Strata lands on Apple Silicon, currently available by building from source; the official announcement is excited, saying “Now on Mac,” and the release page even specially made a macOS-window-style card to celebrate the moment.
Second, models and quantization: Qwen3.6-35B-A3B is newly added, with 4-, 6-, and 8-bit versions; the installer adds Unsloth’s UD-Q4, UD-Q5, UD-Q6, and UD-Q8 quantization packages and the NVFP4 format. NVFP4 is limited to NVIDIA GPUs; it is NVIDIA’s 4-bit floating-point quantization format and is especially favored on next-generation GPUs; UD is Unsloth’s dynamic quantization package series, with four bit widths letting users balance speed and quality themselves.
Third, optional acceleration: with the new –adapt-async parameter enabled, code and document-related work is another 7% to 10% faster, and it also brings background cache updates, making it especially friendly to the RTX 5070.
Fourth, platform improvements: AMD multi-GPU and decode speedups, Intel A-series GPU support (thanks to community member aslater3 for submitting four PRs in one go), and on Windows, the STRATA_PIN_BY_NEED=1 environment variable makes prompt processing another 4.4% faster.
What it means for Mac players
Apple Silicon support means M-series Macs have finally entered the fray; previously, local inference tools like these mostly catered to Windows and Linux users with AMD/NVIDIA GPUs. As for quantization package options, NVFP4 is an NVIDIA-exclusive format, while Macs use UD-series quantization packages. Another piece of good news is the unified memory architecture of Apple chips: the CPU and GPU share a large block of memory, giving it an inherent advantage for fitting large-weight models. The official addition of Mac support at this time is well-timed.
The significance of the Mac version is far greater than “one more platform.” The Mac has long been a stronghold for developers and coding agent workflows, yet for a long time this wave of local models has been relatively cold toward Mac users. Most tools are uniformly tied to discrete GPUs on Windows and Linux, leaving M-series Macs to watch from the sidelines (they can run models, but decoding is usually very slow). After Strata added support, Macs with large unified memory can hold far more weights than 12GB graphics cards. Combined with local APIs compatible with OpenAI and Anthropic, this amounts to moving the entire development workflow onto your own machine, with data still never leaving home. For the entire Apple ecosystem, this path also adds an option beyond official solutions: if you want 125B-class, fully offline models, you no longer have to leave the Mac.
Community speed is its moat.
Strata has turned running flagship models on consumer-grade graphics cards from a pointless flex into an everyday thing. Three months ago, to run a model on the level of Qwen3.8-Flash-Next, most people’s only options were renting a cloud GPU or waiting for the big vendors to show mercy; the current meta answer is your own gaming card plus a cup-of-coffee donation. Readers who want in should first check the official hardware list to confirm their graphics card is supported. There are ready-made one-click installers for Windows and Linux; Mac currently requires building from source. Model sizes range from Q2_0 to IQ3_S, chosen according to memory, and 64GB of RAM is enough to max out the full spread. Downloading, installing, and getting to your first prompt takes about as long as a cup of coffee takes to go cold. The official project also provides an entire community benchmark database, so before buying a card or choosing a model size, you can check other people’s scorecards first.
Source: KOCPC Chinese