Bought a new laptop recently? Chances are that it's equipped with an NPU, a chip you're almost certainly not using. The NPU, or Neural Processing Unit, is a block of silicon that runs AI workloads at a fraction of the power a CPU or GPU needs. Roughly 2W compared to well over 100W that standard laptop discrete graphics need to do the same job. While they're not yet found in all laptops, they're now pretty much standard in mid-range and premium machines.
Advisory firm Gartner reckons that AI-capable PCs, like these NPU-equipped laptops, make up more than half of total global shipments. The reason we're seeing this isn't because buyers want or need them, but to align with certification requirements. Back in 2024, Microsoft pretty much dictated that laptop makers will need to hit 40 TOPS (the measure of how quickly an NPU can chew through AI math) of dedicated NPU performance with new machines, or risk losing access to new Windows AI features. NPU hardware is now flying off the shelves as a result, but software is lagging behind.
An NPU's job is to accelerate inference. That's the "run this model" phase rather than the GPU-powered "train this model" phase. Neural networks rely heavily on two kinds of computation to do this: matrix multiplication and nonlinear activation functions.
