Currently, large language models execute inference tasks in two distinct phases: the first, Prefill (prompt preprocessing), devours massive GPU compute, while the second, Decode (token-by-token generation), is a battleground of memory bandwidth. Nvidia’s Blackwell GPUs excel at the former, and Apple Silicon’s unified memory architecture dominates the latter. So what if you connected them together, letting each focus on what it does best—would that beat any single machine running alone? Famous tech YouTuber Jeff Geerling decided to answer that question with action. Over several months, he linked an Nvidia DGX Spark (codenamed GB10) with a Mac mini M4 Pro and a Mac Studio M3 Ultra through the Exo open-source framework, testing “heterogeneous disaggregated inference” (disaggregated prefill/decode).
What is Disaggregated Inference?
Disaggregated Inference is a computing architecture that splits the inference workload across multiple hardware resources instead of running the entire model on a single device. Instead of deploying a complete AI model on one GPU or server, different stages of inference (such as preprocessing, model layers, or postprocessing) are distributed across separate compute units, often in different physical locations or even across data centers.
This approach allows organizations to scale individual components independently, optimize resource utilization, and reduce latency by placing computation closer to where data originates.
Before diving deep into the experiment, let’s quickly clarify a concept. AI model inference is divided into two stages. The first stage is called Prefill (also known as Prompt Processing), where the model reads and understands your entire prompt and compresses it into a KV Cache. This stage is GPU compute-intensive (compute-bound), where GPUs like Blackwell and H100 that have a large number of Tensor Cores perform the best.
The second phase is called Decode, where the model generates responses token by token. This phase is bottlenecked by memory bandwidth, since generating each token requires reading the entire model’s weights from memory. Apple Silicon’s unified memory architecture is well-suited for this: the M4 Pro delivers 273 GB/s, and the M3 Ultra goes even higher at 819 GB/s of memory bandwidth.
This isn’t just academic talk. Companies like DeepSeek and ByteDance have already deployed disaggregated inference at scale in data centers, assigning prefill and decode to different machines, each optimized separately—which is also one of the reasons inference costs have continued to drop in recent years. But moving this concept to a typical consumer’s desktop is another matter entirely.
Experiment Configuration: Spark as Prefill, Mac as Decode
Jeff chose these machines he had on hand:
- MSI Edge Expert (equivalent to DGX Spark / GB10)Equipped with Nvidia Blackwell GPU and 128GB unified memory, optimized for compute-heavy prefill
- Mac Mini M4 ProFirst generation: 64GB memory, 273 GB/s bandwidth, excels at decoding
- Mac Studio M3 Ultra(Added in Phase 2): 512GB memory, 819 GB/s bandwidth
The choice of these two architectures wasn’t accidental—DGX Spark has Blackwell’s powerful GPU compute, but its memory bandwidth falls far short of Apple Silicon, while Mac is exactly the opposite. What Jeff wanted to test was whether the intuition of “letting each machine do what it does best” actually holds up.
On the software side, they’re using the open-source project Exo, which already has experimental disaggregated prefill/decode support, though it’s never been verified on real hardware. Jeff had Claude Code write a lot of code, SSH’d into two machines to compile the Rust networking layer, compiled Blackwell’s CUDA kernels, and spent several days building the entire environment by compiling MLX’s Metal shaders from source on Mac.
Stuck: mDNS Bug and the Network Nightmare
The biggest technical challenge is networking. Exo uses mDNS to discover peers on the network, but libp2p’s mDNS has a bug on macOS—the two machines can never see each other.
Jeff spent hours trying: direct Ethernet cable, USB adapter, modifying the Rust networking layer, Thunderbolt cable—all failed. Finally he used tcpdump to catch the root cause: it turned out that libp2p’s mDNS implementation had issues on macOS.
The solution turned out to be surprisingly simple—have GB10 actively dial Mac Mini instead of waiting to be discovered via mDNS. After setting an environment variable, the connection was immediately established.
Network Upgrade: Thunderbolt 5 External Mellanox 50Gb NIC
Even when connected, the bottleneck of 2.5Gb USB Ethernet immediately becomes apparent. With a 25,000 token prompt, GB10 calculates the KV cache in less than a second, but transmitting it over the 2.5Gb network takes 25 seconds. 96% of the time is spent on the network.
To solve this problem, Jeff pulled out his Thunderbolt 5 external enclosure, which has a Mellanox ConnectX4 50Gb network card inside. macOS has had built-in drivers since 2019, so it works plug-and-play. After connecting through a Microsec CSR812 switch, KV Cache transfer time was reduced by about 30%. This also highlights the biggest physical limitation of disaggregated inference in consumer environments: you need a fast enough local network, otherwise all the time saved on prefill will be eaten up by the network.
Round 1: Mac Mini + Spark Benchmark
After switching to the non-thinking model Llama 3.1 8B, the data started making sense:
- GB10 Solo PrefillUp to nearly 1,800 tokens/sec
- Mac Mini Decodes Independently：52 tokens/sec
- Time to First Token in Disaggregated Mode: 2.4 seconds, nearing GB10’s 2.3 seconds
- Disaggregated Decode: 34 tokens/sec (slower than standalone Mac Mini due to KV Cache injection overhead)
The time to first token nearly matches GB10 level, meaning you get GB10-grade prefill speed paired with Mac-grade decode. While decode takes a slight hit from KV Cache transmission overhead, the overall direction is correct.
Round 2: Switching to Mac Studio M3 Ultra
Jeff’s real goal is the Mac Studio M3 Ultra, with 512GB of memory and three times the memory bandwidth.
Same test for Llama 3.1 8B:
| Indicator | Spark Alone | Mac Studio alone | Disaggregated |
|---|---|---|---|
| Prefill 4K (tok/s) | 1,585 | 1,420 | 1,584 |
| Decode (tok/s) | 14 | 106 | 84 |
While the disaggregated decode at 84 tok/s is slightly lower than Mac Studio standalone at 106 tok/s (KV Cache injection overhead consumes about 20%), it’s already 6x faster than Spark standalone at 14 tok/s. The 50Gb network link only adds about 18ms of overhead, which is practically negligible. The bandwidth hypothesis is fully validated on the 8B model—Mac Studio decoding is 8x faster than Spark, and the disaggregated approach successfully preserves this advantage.
Upscaling Models: 32B vs 27B Test
Next, Jeff scales up the model. Qwen 2.5 32B (BF16 on Spark, 4-bit on Mac Studio):
- Spark solo prefill: 875 tok/s
- Mac Studio solo prefill: 356 tok/s
- Disaggregated: 792 tok/s (tracking Spark level)
Gemma 2 27B also follows a similar pattern: Spark 779, Mac Studio 379, Disaggregated 722.
Interestingly, when scaling the model from 8B to 27B/32B, Mac Studio’s advantage on decode shrinks: at 8B, Mac Studio’s decode is 8x faster than Spark; by 27B-32B, the gap narrows to just 1.25-1.3x. This is due to the architectural characteristics of larger models—Gemma’s sliding window attention and Qwen’s kernel fusion reduce bandwidth requirements during the decode phase, improving Spark’s relative decode performance.
Honest verdict: Cool, but a single RTX Pro 6000 might be stronger.
After presenting all the data, Jeff gave this impressive summary:
「作為異質推論的概念驗證，這真的很酷。如果你已經擁有這兩台機器，那很好，試著多榨一點 Juice 出來吧。但現實是，DGX Spark 和 Mac Studio 都不便宜。如果我要花這種錢買新的桌上設備，我寧可去買一張 RTX Pro 6000，圍繞它打造一台主機。」
This isn’t just hype. A single RTX Pro 6000 (Blackwell workstation GPU) has 6x the memory bandwidth and 3.5x the compute power of GB10, and will very likely crush the combined performance of these two machines on both prefill and decode.
Anyone interested can go check out Jeff’s crazy experiment this time, though the results weren’t quite what was hoped for.
EXO and Nvidia are also doing the same thing
Jeff’s experiment is far from unique. The open-source framework Exo has long championed the vision of turning any device into an AI inference cluster—desktops, laptops, servers, and even smartphones can all join the collaborative mesh.
EXO Labs’s own testing shows that using two DGX Spark units paired with one M3 Ultra Mac Studio achieves 2.8x overall speedup on Llama 3.1 8B, matching Spark’s prefill speed while maintaining Mac Studio’s fast generation time.
Nvidia itself has also recognized this trend. Its upcoming Rubin CPX platform will use compute-intensive Rubin CPX processors for prefill, while standard Rubin chips handle decode with massive HBM3e bandwidth—exactly the same principle that Exo and Jeff demonstrated on consumer hardware.
Conclusion
While Jeff Geerling’s experiment honestly acknowledged that “a single RTX Pro 6000 might be more practical,” it demonstrated that disaggregated inference can fully operate on consumer-grade hardware, even if it requires working around mDNS bugs, building a custom Thunderbolt external network card, and compiling various kernels.
More importantly, it shows that the AI hardware ecosystem is moving toward a path where the “single superchip” is no longer the only solution. In the consumer market, we may soon see more of these heterogeneous computing combinations: buying a Blackwell GPU device for prefill, paired with an Apple Silicon device for decode—each leveraging its strengths. Nvidia’s Rubin architecture has already embraced this concept at the data center level, and it’s only a matter of time before it trickles down to the desktop.
But if you ask Jeff for advice? “Buy an RTX Pro 6000, thanks.”
Source: KOCPC Chinese