Artificial intelligence chips: how they work and what the names mean

Artificial intelligence chips are processors, or specialized parts of processors, used to run machine learning calculations. Some were designed for neural networks; others, including GPUs and CPUs, also handle many kinds of computing. They execute the operations behind an AI model rather than storing intelligence as a physical property of the chip.

The names can be confusing because they describe different things. GPU names a processor architecture, NPU describes a neural processing role, and ASIC describes a chip designed for a particular application. A TPU is both an AI accelerator and an ASIC. These categories overlap.

This guide explains the technology and terminology. For a workload-based purchasing decision, use our practical guide to AI accelerators alongside the model's software requirements.

What AI chip design changes

Many AI workloads, including deep learning, involve multiplying and adding large collections of numbers. Specialized AI chips can accelerate this work through parallel processing, specialized arithmetic units, and keeping data close to those units. The useful combination depends on the model and how it runs.

Specialization has limits. A chip might execute a supported matrix operation efficiently while another part of the application waits for data, runs on a CPU, or needs an unsupported operation. The full program determines whether the accelerator helps.

It is also inaccurate to divide the world into sequential CPUs and parallel AI chips. Modern central processing units (CPUs) have multiple cores and vector instructions, and some include matrix acceleration. Intel's AI processor overview includes CPUs as well as discrete and purpose-built accelerators. A separate accelerator is not a requirement for every AI application.

AI algorithms remain software: the same architecture can execute many supported models. AI developers need to understand what the hardware accelerates, but they do not have to design their own chips to use it. AI chip design is a separate engineering task from model development, and using AI tools to help design a chip is a different subject again.

How AI chips work: from input to output

1. The application turns an input into numbers

A photo, audio recording, or text prompt must be represented numerically before a neural network can process it. Software prepares the input in the form the model expects. A tensor is simply an array of values, potentially with several dimensions; the word does not identify a particular brand of chip.

For an image model, those values might describe pixels. For a language model, a tokenizer first turns text into tokens, which the model represents using numerical vectors. Data preparation and scheduling can involve the CPU even when an accelerator handles most neural network calculations.

2. Processing units combine inputs with model weights

Weights are numerical parameters learned during training. One small calculation might multiply the inputs 2 and 3 by weights 0.5 and −1, then add the products: (2 × 0.5) + (3 × −1) = −2. Real neural networks perform much larger collections of related operations, along with transformations such as activation functions and normalization.

Matrix units perform many of these multiply-and-add operations together. They do not need to reproduce the physical structure of a brain. Google's TPU architecture documentation, for example, describes connected arithmetic units arranged to pass values through a matrix-processing array.

3. Memory and software keep the work moving

The hardware needs model weights, input values, and intermediate results at the right time. Local storage on the chip can reduce repeated transfers, while larger memory holds data that will not fit there. A fast arithmetic unit is of little use while it waits for the next values.

The compiler and runtime translate model operations into work the device supports. They may combine operations, choose numerical formats, and decide where each part executes. Results then return to the rest of the application, which might display a classification, generate text, or trigger another processing step.

AI chip types and the categories they overlap

Term	What it describes	Important qualification
CPU	A general-purpose processor used for application logic, data preparation, and AI computation	Can run models itself; capabilities vary by processor and software
GPU	A processor architecture suited to substantial parallel work	Used for training and inference, as well as graphics and other calculations
NPU	A neural processing unit optimized for neural network operations	May be part of a larger chip; the name does not define one standard architecture
TPU	Google's Tensor Processing Unit family	A specific family of AI ASICs, not a generic name for every tensor-capable chip
ASIC	An application-specific integrated circuit	A broad design category; a purpose-built AI ASIC can still run different supported models
FPGA	A field-programmable gate array with configurable logic	Can implement custom processing pipelines; configuration requires suitable engineering tools

CPUs and GPUs do different parts of the job

A CPU runs the operating system and general application code. In an AI service, it can handle request routing, tokenization, storage access, and postprocessing. It may also be a practical inference engine when the model and response-time requirements fit its capabilities.

Graphics processing units provide many execution resources for parallel work. Modern GPU designs may also include specialized matrix hardware. Their usefulness extends beyond training: the same GPU can serve a model, generate images, or perform scientific calculations when compatible software is available. Intel's CPU and GPU comparison explains the architectural distinction without treating either processor as universally superior.

An NPU may already be inside a device

A neural processing unit can be integrated with CPU and GPU components in a system-on-chip, or SoC. That is common in phones and laptops, but NPUs are not restricted to consumer devices. IBM's NPU overview describes both integrated consumer implementations and separate data-center hardware.

Having an NPU does not mean every AI feature uses it. The application, model operations, supported formats, and runtime determine which compute units are selected. Nor does the NPU label guarantee that a model fits in memory.

ASICs and FPGAs describe different kinds of specialization

An ASIC has hardware designed for a particular application or class of work. Its circuits are fixed after manufacturing, but that does not mean its model weights are fixed or that it can execute only one neural network. The supported operations and software interface determine its flexibility.

An FPGA allows logic and connections to be configured after manufacturing. That can support a custom data path, numerical format, or connection to sensors. It is not the same development process as installing a different GPU program. Our FPGA versus GPU guide covers the engineering trade-offs in detail.

Training and inference use AI hardware differently

During training, a model processes examples, compares its output with an objective, and updates parameters. A typical neural network training run includes a forward pass, gradient computation, and an optimizer step. Fine-tuning also changes parameters, though it may update only a subset.

Inference uses a trained model to produce an output from a new input. It normally does not update the model's weights. That does not make inference a small or uniform job: serving a large model to many users has different demands from recognizing one object on a camera.

The labels “training chip” and “inference chip” describe intended uses, not an absolute boundary. AWS explicitly positions Trainium for both training and inference, while its Inferentia family focuses on inference. Product generations and software support matter more than assuming a task from the name alone.

Why memory matters as much as arithmetic

Memory capacity determines how much data can be held. Memory bandwidth describes how quickly it can be transferred. A device can have enough capacity for a model yet spend much of its time moving weights rather than performing calculations.

For a simple illustration, seven billion parameters stored at two bytes each require 14 billion bytes, or 14 GB in decimal units, for the weights alone. At four bits per parameter, the raw weight calculation is 3.5 GB. Neither figure is a complete runtime memory requirement: quantization metadata, working buffers, activations, and attention caches can add to it. Training also needs additional state.

Lower precision can reduce storage and data movement, but the chosen model, kernels, and hardware must support it. Quality needs to be checked after conversion. A smaller checkpoint is not proof of faster execution or unchanged output quality.

NVIDIA's performance guide distinguishes limits imposed by arithmetic, memory bandwidth, and latency. This is why two models on the same accelerator can achieve very different fractions of its advertised peak performance.

Software determines which capabilities an application can use

An AI model is not automatically portable just because two chips support the same numerical format. Check the framework version, runtime, supported operators, drivers, and model conversion path. A missing operation may fail, use a slower alternative, or require execution on another processor, depending on the software.

For example, AWS Neuron supplies tools for AWS's purpose-built AI hardware. That software layer is part of the deployment, not a detail that can be ignored after selecting a chip.

Record the environment when comparing performance. A result produced with one runtime, batch size, or model revision may not reproduce under another. Before changing hardware, check whether the existing system is waiting on storage, data preparation, or a software configuration problem.

AI chip makers: design, manufacturing, and packaging

NVIDIA and AMD design GPU products used for AI. Google's TPU and AWS's Trainium and Inferentia are examples of purpose-built cloud accelerator families. Apple and Intel integrate neural processing capabilities into consumer computing platforms. These companies address different markets; this is not a ranking or a complete supplier list.

AMD Instinct, for example, is a GPU family for data-center workloads that includes AI training and inference. Competing with NVIDIA does not necessarily mean offering an interchangeable device: memory, software, interconnects, and deployment options need separate comparison.

The AI chip industry includes distinct activities: chip design, wafer fabrication, memory production, packaging, and cloud operation. AI chip designers can create a processor that another company manufactures. TSMC describes its business as a dedicated semiconductor foundry, manufacturing products for customers.

Logic chips perform computation and control operations; memory chips store data. AI systems need both functions, whether components sit in separate packages or close together in a larger assembly. Semiconductor manufacturing and system assembly turn these designs into usable products, so the name on an accelerator does not identify every company involved in its production.

Packaging also matters. A finished processor package can contain more than one silicon die, and advanced packages can connect multiple components closely together. TSMC's 3DFabric technologies illustrate this use of stacking and advanced packaging. The word “chip” in a product announcement may therefore refer to something more complex than one piece of silicon.

The semiconductor supply chain and the service provider both affect availability. A processor announcement is not evidence that a specific server configuration is shipping or that a cloud region has capacity. Confirm the product, supported software, delivery date or instance availability, and support terms.

Where AI chips are used

Data-center AI applications

In data centers, accelerator chips can support AI model training, run recommendation systems, process images, and serve generative AI models. The system around the chip provides networking, storage, power, cooling, and recovery when hardware or software fails.

Edge AI and connected devices

With edge AI, processing happens on or near the device collecting data. Examples include speech processing on a phone, visual inspection beside a production line, and image recognition on a camera. Power limits, heat, connectivity, and response time can matter more than maximum processing power. An application that needs to work without an internet connection must have the model and required processing available locally.

Apple's research on deploying models on the Neural Engine describes Core ML using a combination of CPU, GPU, and Neural Engine resources. That is a useful example of several processing engines cooperating within a device.

Local execution can reduce the need to send inputs to a remote service, but the presence of an AI chip is not a privacy guarantee. An application can still upload data, logs, or results. Check the actual data flow and settings rather than inferring privacy from the hardware label.

How to interpret AI hardware performance claims

TOPS and FLOPS describe rates of operations under specified conditions. They do not directly tell you how quickly an application responds or whether its output meets your quality requirements. Precision, sparsity assumptions, batch size, and what is counted all affect the comparison.

For large language models, consider time to first token, generation speed, and behavior under concurrent requests. For training, measure progress toward the required quality as well as elapsed time. For a camera, measure response time and detection quality under representative conditions. Our guide to TOPS, FLOPS, and AI performance explains the metrics.

MLPerf Inference results provide standardized workload and scenario context. Read the submission details and system configuration before comparing results; a whole-system benchmark is not a measurement of an isolated chip.

Energy efficiency claims also need a defined unit of work. Compare energy for the same task at acceptable quality, and state whether the measurement covers the accelerator, server, or wider facility. Lower device power does not guarantee lower energy per completed job if execution takes longer.

Accessing AI hardware through Compute with Hivenet

Compute with Hivenet provides GPU and CPU instances for workloads such as model experiments, inference, fine-tuning, and general computing. Renting an instance provides access to existing hardware without purchasing and installing a physical server.

It still requires a compatible model and software environment. Your team manages the operating system, frameworks, dependencies, and workload on the instance. Confirm the available configuration and current price in the console, then test with your data. Access to an accelerator alone does not guarantee a particular speed, saving, or model quality.

Common questions about AI chips

Are AI chips different from regular computer chips?

Some contain specialized units for neural network calculations, but there is no clean dividing line. CPUs can include AI acceleration, GPUs support many non-AI tasks, and a single processor package can combine several types of compute engine.

Does every AI application need a GPU or NPU?

No. Start with the application's requirements and a measured baseline. A CPU may be sufficient. Specialized hardware becomes useful when its supported operations, capacity, speed, or energy characteristics address a real constraint.

Is an AI chip programmed with one model permanently?

Usually not. A fixed hardware architecture can load different model weights and run different supported computations. Changing the software model is distinct from redesigning or reconfiguring the physical circuitry.

How much does an AI chip cost?

First identify what is being priced: a component inside a device, an accelerator board, a complete server, or rented compute time. Those are different purchases. Compare a specific supported configuration and its operating costs instead of applying one price range to the entire category.