What is an NPU and what is it good at?

If you bought a recent laptop, there is a reasonable chance it contains a processor you did not have a few years ago: an NPU.

An NPU, or neural processing unit, is a specialized processor designed to execute neural-network operations efficiently. It handles repetitive mathematical work such as matrix multiplication and other tensor operations that appear throughout machine learning models.

That sounds similar to what a GPU does, and there is substantial overlap. The difference is specialization.

A modern GPU is a highly parallel processor that can handle AI alongside graphics, simulation, rendering, scientific computing, and other workloads. An NPU narrows the problem. It dedicates hardware and software to the operations common in machine learning, often with power efficiency as an important design goal.

On current AI PCs, this usually means running supported AI features locally without keeping the CPU or GPU busy. Intel describes its integrated NPU as the processor for sustained, lower-power AI workloads, alongside a CPU for general-purpose and responsive work and a GPU for higher-throughput tasks. AMD uses a similar CPU, GPU, and NPU arrangement in Ryzen AI systems.

But there is an important complication.

The NPU in a laptop is only one kind of NPU. The same term is also used for edge accelerators and server-class AI processors. Huawei's Ascend platform, for example, supports both training and inference on NPU-equipped servers. So “NPU” describes a broad type of AI processor rather than one particular performance class.

That distinction matters whenever someone tells you that an NPU is faster, slower, cheaper, or more efficient than a GPU. Before comparing them, you need to know which NPU and which workload.

NPU meaning

NPU stands for neural processing unit.

It belongs to the broader family of AI accelerators: processors or processing blocks designed to accelerate machine-learning calculations.

A typical neural network repeatedly applies mathematical operations to large arrays of numbers. These arrays are usually represented as tensors, which is why tensor operations, matrix multiplication, multiply-accumulate operations, convolutions, and related calculations appear so frequently in AI hardware.

General-purpose CPUs can perform these calculations. GPUs can execute them much faster in parallel. NPUs go further by devoting more of their architecture specifically to neural-network execution.

This can make an NPU efficient when the model maps well to the operations, data types, memory layout, and runtime the processor supports.

NPU at a glance

‍

‍

Why do NPUs exist if CPUs and GPUs already run AI?

The answer comes down to specialization.

A CPU needs to do almost everything a computer asks of it. It runs operating-system logic, applications, databases, networking, file operations, control flow, and countless other tasks. Modern CPUs also include vector and matrix instructions that can accelerate machine learning, but AI is only one part of their job.

GPUs occupy a different point on the spectrum. Their architecture is designed around large amounts of parallel work. That happens to map extremely well to the calculations used in neural networks, which helped turn GPUs into the primary hardware for modern deep learning.

NPUs narrow the job further.

They can dedicate more silicon to the calculations, data movement, low-precision arithmetic, and scheduling patterns common in neural networks. That can reduce the energy required to execute supported models, which becomes important when AI needs to run continuously on a laptop or another power-constrained device.

Intel's current Core Ultra architecture makes this division explicit. The company positions the CPU for responsive general work, the GPU for high-throughput workloads, and the NPU for sustained AI at lower power.

The three processors can therefore complement one another instead of competing for every task.

We look at that system-level choice in more detail in CPU vs GPU vs NPU for AI and general compute.

How does an NPU work?

There is no universal NPU architecture, so it is safer to understand the common design ideas than to imagine one standard chip.

Most NPUs contain hardware designed to perform large numbers of neural-network arithmetic operations efficiently. Multiply-accumulate operations are particularly important because neural networks repeatedly multiply values and accumulate the results when processing layers.

Intel's current NPU implementation gives us a concrete example.

Its Core Ultra 200-series NPU contains Neural Compute Engine tiles built around purpose-designed multiply-accumulate engines, along with local memory and processors for handling additional deep-learning operations. Intel's current documentation lists INT8 and FP16 among the inference data types supported by the NPU implementation.

Other NPUs make different choices.

The amount of local memory may differ. The supported numerical formats may differ. Some architectures focus heavily on dense matrix operations, while others add specialized hardware for particular model patterns. The scheduler, compiler, memory system, and way work is distributed across the chip can all change.

This is why knowing that a computer has “an NPU” tells you less than knowing its actual architecture, supported software, memory, and measured performance.

NPUs often use lower-precision arithmetic

Machine-learning models do not always need 32-bit floating-point numbers for every operation.

Using smaller numerical formats can reduce memory requirements and allow more calculations to be performed with the same amount of hardware. INT8, FP16, BF16, INT4, and newer low-precision formats therefore appear throughout modern AI systems.

Client NPUs make extensive use of this idea.

The Intel example above supports INT8 and FP16 inference primitives. AMD's Ryzen AI software similarly includes model quantization as part of its NPU deployment flow and supports deploying pretrained models through ONNX Runtime.

Large language models can take the idea further. Moving a model from 16-bit weights to 8-bit or 4-bit representations can dramatically reduce how much memory is required to store it.

That is one reason quantization has become so important for local AI. Our guide to LLM quantization explains the trade-offs between formats such as INT8 and INT4 and why smaller models do not automatically run faster.

What is an NPU good at?

NPUs make the most sense when the workload is neural-network-heavy, supported by the available software, and benefits from specialized execution.

On client devices, that often means AI jobs that run frequently or continuously.

Video and camera processing

Background segmentation, subject tracking, auto-framing, eye-contact correction, image enhancement, denoising, and other camera features can involve neural networks running throughout a call or recording session.

Those workloads are a good match for an integrated NPU because they are repetitive and persistent. Moving them away from the CPU or GPU can leave those processors available for other work.

Speech and audio processing

Noise suppression, speech recognition, transcription, voice activity detection, and other audio features can also run locally on supported NPUs.

Again, sustained processing matters. A small neural network executing throughout a meeting is different from a GPU workload that finishes in a few seconds and stops.

Computer vision

Object detection, image classification, segmentation, pose estimation, and similar models have long been targets for specialized AI accelerators.

At the edge, these models can run in cameras, gateways, industrial systems, and other devices without continuously sending raw data elsewhere.

Our guide to edge AI hardware looks at where NPUs, GPUs, CPUs, and specialized accelerator cards fit when latency, connectivity, privacy, or power consumption constrain the deployment.

Generative AI

Modern NPUs are increasingly capable of running generative models.

Microsoft's current Windows ML stack can execute local ONNX models across NPUs, GPUs, and CPUs using hardware-specific execution providers. AMD's Ryzen AI software supports generative workloads including large language models and image generation through its NPU and integrated GPU stack.

The important word is supported.

A neural processing unit is not a magic socket into which any AI model can be dropped. The runtime, model architecture, operators, numerical precision, and available memory still have to fit.

Can an NPU run an LLM?

Yes. Some current NPUs can run large language models locally.

MLCommons' current MLPerf Client benchmark includes NPU acceleration paths and LLM workloads based on models such as Llama 3.1 8B Instruct and Phi 3.5 Mini Instruct. The benchmark supports current AMD, Intel, and Qualcomm NPU platforms alongside GPU configurations.

That tells us something important: local LLM inference on an NPU is no longer merely a demonstration.

It does not tell us that every NPU can run every LLM.

Model size remains a hard constraint. So does memory. The runtime needs implementations for the operators the model uses, and the model often needs to be quantized or otherwise optimized for the target processor.

MLPerf's own client benchmark uses quantized LLM weights because full-precision models are often too large for client systems. It also notes that larger experimental workloads may fail on systems without enough memory.

Context length adds further pressure because the model must maintain growing state as it generates text.

For a small, supported model that needs to run privately on a laptop, an NPU can therefore be useful.

For a much larger model, long context, substantial concurrency, or production-scale serving, the balance often moves toward GPUs or other server-class accelerators. Our NPU vs GPU comparison examines that boundary in more detail.

An NPU does not work by itself

Hardware is only half of the story.

Your application needs a software path from the model to the NPU.

One common route is ONNX Runtime. Its Execution Provider architecture lets the runtime assign parts of an ONNX model to hardware-specific acceleration libraries for CPUs, GPUs, FPGAs, and NPUs. Current providers include Intel OpenVINO, AMD Vitis AI, Qualcomm QNN, and Huawei CANN.

That abstraction helps developers target different hardware without rewriting an entire application around every accelerator.

The underlying hardware still matters.

If an execution provider cannot run an operation used by the model, the workload may need to be split between processors or fall back to another device. ONNX Runtime warns that model partitioning caused by unsupported operators can hurt performance on mobile and edge systems.

Intel's OpenVINO provider illustrates how heterogeneous execution can work. It can target CPU, GPU, or NPU hardware and can distribute workloads among several devices when necessary.

Windows is moving in a similar direction. Windows ML now manages execution providers for supported hardware so applications can access CPU, GPU, and NPU acceleration through the same local inference framework.

This makes NPUs easier to use, but it does not eliminate hardware differences.

The software stack may matter more than the NPU specification

Suppose two processors advertise similar AI performance.

One supports your model directly. The other requires converting several operators, changing the quantization scheme, using an unfamiliar compiler, and falling back to the CPU for part of the graph.

The specification sheet will not capture that difference.

Software support influences:

• Which model architectures run
• Which operators are accelerated
• Which numerical formats are supported
• Whether dynamic shapes work
• How models are compiled
• Whether custom operations are possible
• How easily performance can be profiled
• What happens when an operator is unsupported
• Which operating systems and frameworks can target the device
• How much engineering work is required when the model changes

AMD, for example, lets developers start from models trained in PyTorch or TensorFlow and deploy them through ONNX Runtime to supported Ryzen AI hardware. Huawei uses its CANN stack and a torch_npu integration to adapt PyTorch workloads for Ascend hardware.

Huawei even provides migration tools because GPU-oriented PyTorch code cannot simply be assumed to run unchanged on Ascend NPUs. Its documentation explicitly points to architectural differences between GPUs and NPUs as the reason migration is required.

That engineering cost belongs in any serious hardware comparison.

It is one of the main subjects we will examine in specialized AI accelerators vs GPUs for inference.

Are all NPUs built into laptops?

No.

The current wave of AI PCs has made the term familiar, but neural processing hardware exists in several forms.

Integrated client NPUs

These sit inside a system-on-chip or processor package alongside the CPU and GPU.

Their main advantages are proximity to the rest of the system, low power consumption, and the ability to execute AI locally without needing a discrete accelerator.

Intel Core Ultra and AMD Ryzen AI are current examples.

Mobile and embedded NPUs

Phones, tablets, cameras, vehicles, and embedded systems can include dedicated neural-network acceleration.

Power, thermal limits, physical size, predictable latency, and offline execution become especially important here.

Edge accelerators

Some NPUs appear as dedicated modules or accelerator hardware inside industrial and edge systems.

A camera network, robotics system, manufacturing line, or remote sensor deployment may need considerably more AI performance than a small integrated NPU while still operating outside a conventional data center.

Data-center NPUs

The term also reaches server hardware.

Huawei's Ascend platform includes NPU-equipped systems for training and inference, together with its CANN development stack and PyTorch integration.

That makes statements such as “NPUs are only for laptops” or “NPUs cannot train models” inaccurate.

The hardware class needs to be specified.

Can an NPU train AI models?

For most client NPUs, training is not the main job.

AMD's current Ryzen AI development flow begins with a pretrained model before optimization and deployment to the NPU. Intel likewise describes its client NPU primarily as an inference and deep-learning acceleration engine for power-sensitive PCs.

Server-class hardware changes the answer.

Huawei documents PyTorch training on Ascend NPU servers, including model migration and separate software packages for training and inference.

Other accelerator families make their own trade-offs.

This is why we separate the processor taxonomy from the workload question in training vs inference hardware. Hardware that is excellent for repeatedly serving a fixed model does not automatically make the best development or training environment.

How is NPU performance measured?

The number you will encounter most often is TOPS, or trillions of operations per second.

TOPS estimates the theoretical number of operations a processor can perform each second under specified conditions.

Intel, AMD, Qualcomm, Microsoft, and PC manufacturers now use TOPS frequently when describing AI PCs and their NPUs.

It is useful as a hardware specification. It is a poor universal benchmark.

A TOPS figure depends on factors such as numerical precision, whether sparsity is counted, what qualifies as an operation, and which part of the processor is being measured. Intel's current AI PC documentation, for example, distinguishes the TOPS contributed by the NPU, GPU, and CPU when calculating total platform AI performance.

You therefore cannot safely compare two devices by dividing one TOPS number by another.

A 50-TOPS NPU is not automatically twice as fast as a 25-TOPS NPU for your model. A GPU advertising far more AI TOPS is not guaranteed to complete your application proportionally faster either.

Software, memory bandwidth, precision, operator support, model architecture, thermals, and utilization all intervene between theoretical peak performance and useful output.

We cover that measurement problem directly in TOPS vs FLOPS and the AI performance metrics that matter.

For real hardware decisions, standardized workload benchmarks are more useful. MLPerf Client, for example, measures complete LLM workloads on supported NPU and GPU execution paths rather than comparing peak arithmetic specifications alone.

The same principle guides our own Hivenet benchmarks: measure the workload rather than trying to infer application performance from a single specification.

NPU vs CPU

A CPU is a general-purpose processor. An NPU is specialized for neural-network calculations.

That specialization can give the NPU better performance per watt on supported AI models, while the CPU remains much more flexible.

The CPU still handles operating-system work, control logic, application code, preprocessing, data movement, networking, databases, and many operations around the AI model itself.

An application running on an NPU therefore still needs a CPU.

The interesting question is which parts of the workload should run where.

NPU vs GPU

An NPU specializes more aggressively in machine learning. A GPU offers broader high-throughput parallel compute.

For a sustained, supported AI task on a laptop, the NPU may offer the best efficiency.

For large models, model development, training, fine-tuning, high-throughput inference, rendering, simulation, or workloads that change often, a GPU generally gives developers a broader operating envelope.

Some applications use both.

For the complete comparison, including memory, power, cost, software support, training, inference, and local LLMs, see NPU vs GPU for AI workloads.

NPU vs TPU, ASIC, and other AI accelerators

An NPU is part of a much larger hardware family.

Google's TPU is another specialized processor designed around machine-learning workloads. ASICs can be designed for even narrower purposes. FPGAs trade fixed hardware for reconfigurability. AI accelerator cards add dedicated processing to servers or edge systems.

The names become confusing because these categories overlap.

Some NPUs are ASICs in the broader engineering sense. Some vendors use proprietary names for hardware that performs work similar to what another vendor calls an NPU. “AI accelerator” is therefore often the safest umbrella term.

Our practical guide to AI accelerators compares these categories by architecture and workload rather than trying to turn vendor terminology into rigid boundaries.

What are the limitations of an NPU?

The advantages of specialization come with constraints.

Model compatibility

A model may contain operations the NPU or its runtime cannot accelerate.

That can force part of the model onto the CPU or another accelerator and change performance substantially.

Memory

Integrated NPUs usually operate within the memory architecture of the surrounding system rather than carrying large pools of dedicated VRAM or HBM.

That can limit large models even when the NPU has sufficient arithmetic capability.

LLMs make this particularly visible. Model weights, runtime buffers, and context-related state all consume memory.

Software support

The hardware needs drivers, runtimes, execution providers, compilers, model-conversion tools, framework integration, and debugging support.

A newer accelerator with impressive specifications may still be less useful than older hardware with a mature software environment.

Precision support

An NPU may be optimized around particular numerical formats. A model that expects a different precision may need conversion or quantization.

Flexibility

Specialized hardware works best when the workload resembles what the designers expected.

Research and development are less predictable. New model architectures appear, operators change, frameworks evolve, and custom kernels may be required.

That uncertainty often favors more programmable hardware.

Comparability

“NPU” tells you almost nothing about performance by itself.

A small integrated accelerator and a server-class Ascend processor can both be called NPUs while belonging to completely different performance, memory, power, and deployment classes.

Do you need an NPU?

For many people, the answer is still no.

An NPU becomes useful when you have software that can take advantage of it.

If you already own a device with an NPU, there is little reason to avoid it when the application supports it. The processor can offload AI work that would otherwise occupy the CPU or GPU.

‍

‍

Buying a machine specifically for its NPU deserves more scrutiny.

Check what applications actually use it, which models and runtimes are supported, how much memory the system has, and what independent workload benchmarks show.

When an NPU stops being the practical answer

Local execution has real advantages.

It can work offline, keep data on the device, avoid network latency, and make efficient use of hardware you already own. For small supported models, that may be exactly what the application needs.

The calculation changes as the workload grows.

You may need more memory than the device provides. The model may not be supported by the local runtime. You may be training or fine-tuning rather than simply running inference. Several users may need the model at once. You may need to test five model families rather than commit to one.

At that point, access to more flexible compute becomes useful.

Compute with Hivenet provides dedicated GPU and CPU instances for teams that want control over their own model, runtime, libraries, and environment. If you need the model endpoint rather than the infrastructure underneath it, the Hivenet Inference API provides managed OpenAI-compatible endpoints instead.

Neither route makes local NPUs obsolete. They solve a different deployment problem.

A useful architecture may run a small model on the device, move larger requests to remote compute, or use both depending on latency, privacy, cost, and model capability. We explore that boundary further in edge AI hardware.

What should you check before choosing an NPU?

Ignore the logo for a moment and start with the workload.

Ask:

1. Which model will run?
2. Does the target runtime support that model on this NPU?
3. Are all important operators accelerated?
4. Which precisions does the NPU execute efficiently?
5. Does the model fit comfortably in available memory?
6. How much preprocessing or postprocessing remains on the CPU?
7. What latency does the application require?
8. How much throughput is needed?
9. Does the workload need to run offline?
10. How much does power consumption matter?
11. Will the model change frequently?
12. What happens when an unsupported operation appears?
13. Is there a benchmark using the actual model or something close to it?

Those questions reveal far more than the advertised TOPS figure.

They also help determine whether the right answer is an NPU, a GPU, a CPU, another specialized accelerator, or a combination of several processors.

FAQ about NPUs

What is an NPU in a computer?

An NPU is a neural processing unit, a processor designed to accelerate machine-learning and neural-network operations. In modern AI PCs, it usually sits alongside the CPU and GPU and handles supported AI workloads that benefit from efficient local execution.

What does an NPU do?

An NPU executes mathematical operations used by neural networks, including large numbers of multiply-accumulate and tensor operations. Applications can use it for workloads such as computer vision, speech processing, image enhancement, generative AI, and local LLM inference when the model and runtime are supported.

Is an NPU the same as a GPU?

No. Both can accelerate AI, but GPUs are broader parallel processors while NPUs are designed more specifically around neural-network workloads. Their capabilities overlap, and some systems use both processors together.

Is an NPU better than a CPU?

For supported neural-network calculations, an NPU can offer much higher efficiency than a general-purpose CPU. A CPU remains necessary for the rest of the operating system and application and can run workloads an NPU was never designed to handle.

Is an NPU better than a GPU?

It depends on the workload. Integrated NPUs can be excellent for sustained, low-power local AI. GPUs generally offer greater flexibility, broader software support, and more capacity for large models, training, fine-tuning, and high-throughput workloads. See our full NPU vs GPU comparison.

Can an NPU run AI without the internet?

Yes. One major use of client NPUs is on-device inference. If the model and application are installed locally, supported AI tasks can run without sending the workload to a remote service. Microsoft and AMD both support local AI execution on current NPU-equipped PCs.

Can an NPU run an LLM?

Some can. Current client benchmarks include NPU execution of models such as Llama 3.1 8B and Phi 3.5 Mini. Model size, quantization, memory, runtime support, and context length determine what is practical on a particular system.

Can an NPU train machine-learning models?

Some NPUs can, but most integrated client NPUs are designed primarily for inference. Server-class NPU platforms such as Huawei Ascend support training through specialized software stacks.

What does NPU TOPS mean?

NPU TOPS means trillions of operations per second and describes a theoretical peak rate of AI arithmetic under defined conditions. TOPS can help compare hardware within a controlled context, but it should not be treated as a direct measure of real application speed.

How many NPU TOPS do I need?

There is no universal target. The model, numerical precision, operator support, memory, runtime, and latency requirements matter more than reaching an arbitrary TOPS number. Benchmark the application you intend to run.

Will NPUs replace GPUs?

They are more likely to share systems with them. CPUs, GPUs, and NPUs address different computational constraints, and current AI PCs already combine all three. Specialized NPUs may take over workloads where their efficiency makes sense while GPUs remain useful for flexible, high-throughput compute.

The useful question is what you want the NPU to run

An NPU is specialized hardware for neural-network processing.

That specialization can make local AI faster and more power-efficient, especially when a supported model needs to run continuously on a client or edge device.

It also creates limits.

The model needs to fit. The runtime needs to support it. The operators need accelerator implementations. The numerical format has to work. And the software around the processor needs to be mature enough that using the hardware does not create more work than it saves.

The NPU label alone therefore tells you surprisingly little.

Start with the model and workload, then examine memory, software support, measured performance, latency, power, and cost.

From there, the choice between an NPU, GPU, CPU, or another accelerator becomes much easier to defend.

Next in the series, we broaden the comparison with a practical guide to AI accelerators, then look at CPU vs GPU vs NPU, specialized AI accelerators vs GPUs for inference, edge AI hardware, TOPS vs FLOPS, and training vs inference hardware.