NPU · Neural Processing Unit
Neural Processing Unit
What is an NPU?
- A low-power processor dedicated to AI computation
- The heart of on-device AI and AI PCs
- How work is divided among the NPU, CPU, and GPU
What an NPU really changes is where AI computation happens. Until now, most enterprise AI features have worked by calling a cloud API, so even summarizing a single meeting meant uploading the audio file to an external server and paying per use. Devices with an NPU finish that computation on the device itself, so the per-call cost disappears and the same feature keeps working in factories or on business trips where the network is unreliable. On the ground, the bigger change is that the 'does the data leave the company?' item that used to trip up every security review no longer exists.
Performance is usually quoted in TOPS (trillions of operations per second), and Microsoft’s requirement for Copilot+ PCs is 40 TOPS or more. But this figure is based on low-precision operations such as INT8, so it’s hard to compare products directly, and perceived speed often depends more on memory capacity and bandwidth than on TOPS. Models that run on a 16GB laptop are generally limited to a few billion parameters, so expecting the same quality as large cloud models will lead to disappointment.
The boundaries of its role are also clear. An NPU is closer to an inference-only chip for running models that are already trained; training or fine-tuning a new model is still the job of GPU servers. So the practical decision isn’t “NPU or GPU” but a task-by-task split: repetitive, sensitive work such as summarizing internal documents, translation, and speaker separation runs on the device, while work that needs a large model runs in the cloud.
The most common misconception is that buying a laptop with an NPU automatically speeds up the AI apps you already use. If the application and drivers don’t support the NPU path, the workload falls back to the CPU, and you end up with expensive hardware that only adds fan noise and battery drain. In practice, it’s not unusual for next year’s budget to be cut because the benefits promised in the purchase proposal were never verified. Another point: on-device hardware doesn’t mean everything is processed on the device. If an app calls the cloud behind the scenes, the security benefit is zero, so you need a process of getting written confirmation from the vendor, before purchase, that the exact app you use runs on the NPU.
Jaryong KooMarketing PhD and leading authority on AI, data analytics, and personal branding
Seungtae ParkAX practitioner who builds AI business development and marketing automation tools and teaches from that hands-on experienceThe concept emerged in the mid-2010s, as the spread of deep learning drove serious development of dedicated accelerators that process AI computation more efficiently than CPUs and GPUs. Neural processing units then began to appear in smartphone application processors, and they have since spread across mobile devices, PCs, and servers.
Consider a hypothetical case. Company A, an auto parts manufacturer with 620 employees, was due to replace 380 laptops issued in 2019. The departments handling design drawings and supplier price lists were under a security policy of 'do not upload internal documents to external AI services,' so their AI usage was close to zero. The IT strategy team spent four months evaluating AI PCs with NPUs.
- The head of the IT strategy team asked each department for 'work we wanted to use AI for but could not because of the policy' and collected 32 requests. Meeting minutes, foreign-language email translation, and spec sheet summaries made up more than half of them.
- The purchasing team was about to choose a model based only on the 45 TOPS figure in the quote. But the team lead said, 'let's first check whether the groupware and PDF viewer we use every day actually use the NPU,' and they asked three vendors for their lists of supported apps.
- We piloted 40 laptops with 40 people across the design and purchasing teams for 8 weeks, keeping 10 existing laptops as a control group and measuring battery life and document-summarization time against the same benchmarks.
- In week 4, complaints came in that “the fan spins but pressing Summarize does nothing.” Checking showed the app was computing on the CPU, and the NPU path was only picked up after the driver and app versions were updated.
- After finalizing the purchase, the team worked with HR to hold three 2-hour hands-on training sessions with 190 participants. They also distributed a one-page guideline to each department that sorted work into 'this document stays on the device, this task goes to the cloud.'
- Eight weeks later, the time spent writing up each meeting had dropped from an average of 35 minutes to 9. The range of documents that could be processed within the security policy also grew, saving about KRW 19 million (≈ US$14,200) a year in external AI subscription costs.