The brain of any modern smartphone is the SoC, an incredibly complex micro-machine. It packages the CPU, GPU, audio and video processing, wireless communication, and power management into a chip the size of a fingernail. As technology advances, SoCs become more and more complex, and one of the latest additions is the NPU, which powers AI features. If your phone is the latest model, it very likely has an NPU inside; even mid-range phones might have one. But what does the NPU do? What difference does it make? Let’s explore.
What is an NPU? How does it make phones smarter?
What is an NPU?
NPUs have been around longer than you might think. An NPU is a neural processing unit, a computing module in a smartphone SoC, similar to a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). Recent Snapdragon, Exynos, Dimensity, and Apple A-series SoCs, as well as some desktop and mobile PC processors from Intel, AMD, and Apple, are equipped with NPUs.
Mobile SoCs have been equipped with NPUs for some time now. Since 2015, Qualcomm has included an AI engine on the Snapdragon 820 (PDF ), this is a combination of hardware and software used to execute AI tasks. However, although they are not yet complete, they are further highlighted today by the hype surrounding AI and the capabilities it brings.
Why do smartphones need an NPU?
The NPU’s job is to accelerate tasks related to AI and machine learning applications, including identifying people and objects in images, generating text and images, converting speech to text, real-time translation, and predicting the next word you might type. In theory, you don’t need an NPU to perform these functions, but it makes the process faster, more energy-efficient, and less dependent on cloud computing. The computation required for AI tasks is very specific, so it makes great sense to optimize a processing unit for these tasks. If you’re curious about this, Michael Stevens of the famous Vsauce channel demonstrates an effective neural network in the YouTube video below; note how it simultaneously performs basic but numerous operations to keep the overall network functioning.
What is the difference between CPU and NPU?
A CPU is a general-purpose unit that can execute one or several complex computations quickly and with high precision. However, AI work requires computing many calculations in parallel, and precision is not as important. Due to the parallel nature of GPUs, GPUs are better suited for this task than CPUs. Even so, just as IBM points outAs such, NPUs still perform excellently due to their efficiency. They can deliver similar AI performance while consuming a fraction of the energy, making them ideal for battery-powered mobile devices.
What are the benefits of on-device AI?
On-device AI is very beneficial for protecting privacy. Whether you provide personal data in the form of voice, text, or images, it never leaves your phone, which also eliminates the chance for bad actors to access it in a data breach.
What makes the TPU in the Google Pixel different?
If you’ve looked at the spec pages for the Google Pixel 9 series, which has been heavily promoted for its AI features, you won’t find any mention of an NPU—that’s because it uses a TPU (Tensor Processing Unit).
Like NPUs, TPUs can accelerate AI computations. The difference is that TPUs and TPU chips are custom-designed by Google, and you can only find them in Google hardware and the company’s data centers. TPUs are optimized for TensorFlow, an open-source software library developed by Google, designed specifically for machine learning and AI applications.
What is NPU TOPS?
Although most phones are equipped with an NPU, some phones execute AI computations faster. TOPS (trillions of operations per second) is a common metric for AI processor performance.Qualcomm explainsThere are two factors that determine the TOPS of an NPU: the operating frequency (clock speed) and the number of MAC operation units it can handle.
The recently released Snapdragon 8 Elite chip is said to deliver a 45% improvement in AI performance over its predecessor, the Snapdragon 8 Gen 3. That’s not much compared to the 1,300+ TOPS offered by the high-end Nvidia RTX 4090 desktop graphics card. Then again, a phone doesn’t draw up to 450W of power like Nvidia’s beast of a GPU. Since TOPS requirements for AI tasks are rarely mentioned, it’s hard to put these numbers into context. However, Microsoft’s Copilot+ AI chatbot requires at least 40 TOPS.
Will NPUs continue to exist?
Given their highly specialized nature, NPUs cannot replace CPUs or GPUs. Instead, their purpose is to improve the efficiency of mobile SoCs by handling AI tasks while saving battery life. As AI becomes integrated into smartphones, we will hear more about NPUs in the future.
Source: KOCPC Chinese