Google Gemma’s official X account recently shared news that excited the developer community: Reddit user Antikythera A self-developed Calibration-Aware Quantization technique was showcased on r/LLMDevs, successfully compressing Gemma 4 to run on iPhone with only about 516MB of active memory, completely offline.
Gemma 4 running on an iPhone with just ~500 MB of RAM!
User Antikythera on r/LLMDevs shared their calibration-aware quantization approach to shrink Gemma 4 keeping speed and accuracy on an iPhone. They demoed it using an offline assistant that manages calendar actions with just… pic.twitter.com/ZVjXvmwB4W
— Google Gemma (@googlegemma) August 4, 2026
Developers successfully ran the Gemma 4 large model offline on an iPhone with 516MB of memory.
To understand why Antikythera’s results are so remarkable, one must first go back to Google’s release on June 5, 2026, not long ago. Gemma 4 QAT(Quantization-Aware Training, QAT) checkpoints. Traditional Post-Training Quantization (PTQ) compresses the model only after training is complete, like forcibly squeezing an already-built house into a smaller space, inevitably damaging the walls and structure. For small models, the quality loss caused by PTQ is especially severe; in models with fewer than 15B parameters, quantization noise is enough to disrupt semantic coherence, leading to a sharp decline in output quality.
QAT takes a completely different approach: it simulates the effects of low-precision computation during the training process, teaching the model to work within the constraints of 4-bit or even 2-bit precision. Technically, the forward pass performs “fake quantization,” mapping weights to integer values and then dequantizing them back to floating point for computation; the backward pass uses a straight-through estimator to bypass the derivative problem of the rounding function, allowing gradients to flow directly to the high-precision master weights. After training, when all weights are permanently converted to low-bit representations, the precision loss is nearly zero.
This time, Google released two types of QAT checkpoints: one is the Q4_0 format for the full model series, and the other is the Mobile QAT format designed specifically for mobile devices. The latter incorporates four key designs for the E2B and E4B edge models: channel-wise quantization (aligned with NPU memory layout), 2-bit compression for the token generation layer, precomputed scaling constants, and custom scheduling for mobile accelerators. With the Mobile QAT format, the memory footprint of the E2B text-only model is reduced to below 1GB.
From 1GB to 516MB: Antikythera’s Extreme Compression
Google’s official QAT has already compressed E2B to under 1GB, but Antikythera’s calibration-aware quantization technology pushes that figure even further to roughly 516MB of active memory. What does this number mean? Most modern smartphones have between 6GB and 12GB of RAM, and a complete AI model takes up less than one-tenth of that space, meaning users can run the model while other phone functions remain virtually unaffected.
More importantly, the Antikythera demonstrates not just that it “can run,” but that it is actually “usable.” From the screenshots he shared, an app called fraQtl can run Gemma 4 in fully offline mode on an iPhone, achieving an inference speed of 13.7 tok/s. When the user said to it in Airplane Mode, “Help me find a time slot open on Thursday for a VC meeting,” the model not only found an opening from 11:30 AM to 12:30 PM, but also directly created the corresponding event in the calendar. Switching to the Calendar app to confirm, the “VC meeting” indeed appeared in the correct time slot.
This on-device model has evolved from a chatbot that simply answers questions into a smart assistant that deeply integrates with the phone system and carries out real tasks. Being fully offline means all data is processed on the device and never sent to any server, a major advantage for privacy-conscious users.
Unsloth’s Contribution: Making Quantization No Longer a Black Box
In the Gemma 4 QAT ecosystem, the open-source community Unsloth plays a key role. They found that directly converting Google’s QAT BF16 checkpoints to the llama.cpp-compatible Q4_0 format results in noticeable accuracy loss—taking the 26B-A4B model as an example, the directly converted version achieved only 70.2% Top-1 accuracy. Unsloth developed a set ofDynamic Quantization Method (Unsloth Dynamic)，improving accuracy to 85.6% while also shrinking the model file by 200MB.
More specifically, with Unsloth’s processing, the E2B model had an average KLD (KL divergence, a metric measuring the difference between the quantized output and the original output) of only 0.00173, a 29-fold improvement over the 0.05109 from direct conversion, while the file size was also 22% smaller. They also introduced the UD-Q2_K_XL quantized version, further compressing E2B and E4B to 2-bit precision, providing an option for devices with extremely limited memory.
The Practical Barriers of On-Device AI
Although the results are encouraging, on-device deployment still faces real-world limitations. The first is the hardware threshold: according to community reports, the E2B model requires at least an iPhone 13 Pro (6GB RAM) or above to run stably, while E4B requires an iPhone 15 Pro (8GB RAM) or equivalent. On devices with less than 6GB of memory, after accounting for the operating system and background apps, there may not be enough space to load the model.
Next is the trade-off between speed and quality. Community benchmark data shows that E2B can reach about 12-20 tok/s on the iPhone 16 Pro, while E4B sits between 8-15 tok/s. Compared to cloud APIs, which often deliver dozens or even hundreds of tok/s, on-device inference still lags noticeably. However, for everyday tasks like calendar management, text summarization, and offline translation, this speed is already practical enough.
Battery and heat dissipation are also factors that can’t be overlooked. Running an LLM on a device is sustained high-load computation, similar in nature to gaming or video export. Extended conversations will noticeably drain battery and cause the device to heat up, while brief usage has minimal impact. Some community developers suggest switching the KV cache to q4_0 format in the app settings, which can significantly improve long-conversation speed at the cost of a small amount of quality.
From Lab to Everyday Life: The Next Step for On-Device AI
Contributions from community developers like Gemma 4 QAT and Antikythera are pushing on-device AI from tech demos into practical deployment. Google’s AI Edge Gallery is now available on the App Store and Google Play, providing a full Gemma 4 model execution environment; the LiteRT-LM runtime behind the MediaPipe LLM Inference API is Google’s inference framework for mobile devices, taking over from the legacy MediaPipe in maintenance mode.
For developers and users in Taiwan, in the near future, AI assistants on mobile devices may no longer need an internet connection to handle everyday tasks—checking calendars, writing summaries, translating documents, and even basic code assistance. Combined with the Apache 2.0 license and open-source checkpoints on Hugging Face, individual developers now have the opportunity to build their own offline AI applications. Just two months after Google DeepMind’s official QAT release, the Reddit community managed to compress memory usage down to 516MB. The pace at which on-device AI has moved from technical demonstrations to everyday practicality has exceeded most people’s expectations.
Source: KOCPC Chinese