TOPS vs FLOPS and the AI performance metrics that matter

One AI processor advertises 50 TOPS.

Another advertises more than 3,000.

Is the second one roughly 60 times faster?

Almost certainly not.

AMD currently lists some Ryzen AI processors with NPUs rated at up to 50 TOPS. NVIDIA lists the GeForce RTX 5090 at 3,352 AI TOPS, alongside 32 GB of GDDR7 memory and 1,792 GB/s of memory bandwidth. Those numbers describe very different processors operating under different constraints. Dividing one TOPS figure by the other does not tell you how much faster either system will transcribe audio, generate an image, run a local language model, or serve an LLM. AMD and NVIDIA publish the underlying specifications.

This is the problem with AI performance numbers.

They are useful.

They are also easy to use outside the context that gives them meaning.

TOPS measures a theoretical rate of operations. FLOPS measures a theoretical or achieved rate of floating-point operations. Neither measures your application.

To understand real AI performance, you eventually have to leave the specification sheet and measure the workload.

What is TOPS in AI?

TOPS means trillions of operations per second.

One TOPS represents one trillion numerical operations every second.

AI chip manufacturers commonly use TOPS to describe the peak computational capability of neural processing units, AI accelerators, and other hardware designed for machine-learning operations.

Qualcomm describes TOPS as a measure of potential peak AI inference performance and explains one common calculation using the number of multiply-accumulate units and their operating frequency:

TOPS = 2 × MAC units × frequency ÷ 1 trillion

The factor of two appears because a multiply-accumulate operation can count the multiplication and addition as two separate operations. Qualcomm also notes that INT8 is a common precision used for reporting NPU inference TOPS. (Qualcomm's guide to AI TOPS)

Already, there are several things to notice.

The number depends on:

• how an operation is counted;
• how many compute units exist;
• processor frequency;
• numerical precision;
• the architecture being measured.

That is why TOPS should be treated as a peak hardware capability under defined assumptions, rather than a prediction of application performance.

What does 40 TOPS mean?

The number 40 appears frequently in discussions about AI PCs because Microsoft uses it as part of the hardware requirements for Copilot+ PCs.

Microsoft's current developer guidance states that many Copilot+ PC AI features require an NPU capable of 40 or more TOPS. (Microsoft Copilot+ PC developer guidance)

That does not make 40 TOPS a universal threshold between a “slow” and “fast” AI processor.

It means 40 TOPS is relevant to a particular class of Microsoft devices and workloads.

A 20-TOPS accelerator may be perfectly capable of running a small computer-vision model.

A 50-TOPS NPU may still be completely unsuitable for a large LLM because the model does not fit into the available memory or the runtime does not support it.

A multi-thousand-TOPS GPU may spend most of its time waiting for memory during a particular inference workload.

The workload decides what the number means.

What is FLOPS?

FLOPS means floating-point operations per second.

The metric describes the rate at which a processor or system can perform floating-point arithmetic.

The prefixes tell you the scale:

• GFLOPS: billions of floating-point operations per second
• TFLOPS: trillions
• PFLOPS: quadrillions
• EFLOPS: quintillions

FLOPS has long been used in scientific computing and high-performance computing because those workloads often depend heavily on floating-point calculations.

AI uses floating point too, particularly during model training, but modern machine learning increasingly relies on several numerical formats.

A training workload might use FP32, TF32, BF16, FP16, or FP8.

Inference may use FP16, FP8, INT8, INT4, FP4, or a mixture of formats.

That means a TFLOPS figure is incomplete until you know which floating-point precision it describes.

TOPS vs FLOPS

The simplest distinction looks like this:

‍

‍

TOPS and FLOPS describe computational capacity.

The other metrics move progressively closer to the work the user actually asked the system to perform.

A FLOP and FLOPS are not quite the same thing

AI papers sometimes make the terminology confusing.

A FLOP can refer to one floating-point operation.

Researchers may therefore describe a model as requiring a certain number of FLOPs to run. In that context, they are talking about an amount of computational work.

FLOPS, with “per second” implied, describes a rate.

So:

model FLOPs = how much arithmetic work is required

while:

hardware FLOPS = how quickly floating-point operations can theoretically or actually be executed

The capitalization is not always used consistently in practice, so check the context rather than relying entirely on typography.

Precision can completely change the number

This is one of the biggest traps in AI hardware comparisons.

An 8-bit integer occupies one quarter of the storage of a 32-bit value.

A 4-bit value needs half as many bits again.

Hardware can therefore fit more low-precision arithmetic into the same area, move less data through memory, and potentially execute more operations in parallel.

That is why the same chip can have radically different peak throughput depending on the numerical format being used.

A processor's:

FP32 throughput

cannot simply be compared with another processor's:

INT8 TOPS

as though the values measured the same work.

They do not.

The precision also affects the model.

Reducing a model from FP16 to INT8 or INT4 can cut memory requirements substantially, but quality must still be tested and speedups depend on whether efficient kernels exist for the chosen format. Our guide to LLM quantization covers that trade-off in detail.

This is especially relevant to the NPU vs GPU comparison. NPUs are often marketed around low-precision AI performance, while GPUs expose several compute paths across different precisions and workloads.

The same processor can have several performance numbers

A modern accelerator is not one arithmetic engine.

A GPU can contain:

• general shader or CUDA execution units;
• Tensor Cores;
• specialized ray-tracing hardware;
• media encoders and decoders;
• caches and memory controllers.

An AI PC processor may contain:

• CPU cores;
• integrated GPU cores;
• an NPU.

Intel, for example, reports CPU, GPU, and NPU AI capability separately on its AI PC platforms because those processors can all contribute different kinds of acceleration. (Intel AI PC)

Adding those figures together may be useful when a software workload can genuinely use all the processors.

It does not mean one model automatically receives the sum of every advertised TOPS figure.

The software has to distribute the work.

Peak performance is not sustained performance

TOPS and peak FLOPS assume that the relevant compute hardware can keep operating close to its maximum rate.

Real processors encounter limits.

Temperature rises.

Power limits intervene.

Memory stalls.

Dependencies appear.

Some execution units sit idle.

Software fails to schedule enough parallel work.

An operation falls back onto the CPU.

A laptop changes performance because it is running on battery.

Peak figures still tell you something about the hardware ceiling.

They do not tell you how close your application will get to it.

This becomes particularly important with edge AI hardware, where sustained thermal and power limits can matter as much as the processor's short-term peak.

Memory can make TOPS irrelevant

Suppose a processor has enormous AI throughput.

Your model requires 40 GB of memory.

The device has 16 GB.

The performance comparison is over before arithmetic speed becomes relevant.

AI workloads depend on the memory system as well as the compute units.

Useful specifications include:

• memory capacity;
• memory bandwidth;
• cache;
• memory type;
• shared versus dedicated memory;
• accelerator interconnect bandwidth.

NVIDIA's current RTX 5090 specifications make the distinction visible. The GPU is rated at 3,352 AI TOPS, but it also provides 32 GB of GDDR7 memory and 1,792 GB/s of memory bandwidth. Those memory figures can matter greatly for models that repeatedly move large weights and KV-cache data.

This is why a model can run faster on hardware with a lower theoretical arithmetic peak if that hardware better fits the model's memory behavior.

LLM inference is often limited by data movement

Large language models repeatedly read model weights from memory while generating tokens.

During the prefill phase, the model processes the input prompt with considerable parallelism.

During decode, output tokens are generated one after another, and memory movement can become a major constraint.

That means adding theoretical compute does not necessarily produce proportional increases in token generation speed.

The processor may already have enough arithmetic capability.

What it needs is faster access to the data feeding that arithmetic.

This is one reason AI accelerator vs GPU comparisons need to examine memory architecture, batching, and serving software alongside peak compute.

TOPS does not equal tokens per second

A TOPS specification tells you nothing direct about how quickly an LLM will generate text.

Tokens per second depends on the entire system.

That includes:

• model architecture;
• parameter count;
• quantization;
• prompt length;
• output length;
• accelerator memory;
• memory bandwidth;
• batch size;
• concurrency;
• serving engine;
• kernels;
• KV-cache management;
• speculative decoding;
• software version.

This is why Hivenet's own RTX 5090 tests measured actual token throughput using a defined Llama workload and vLLM rather than deriving inference speed from NVIDIA's AI TOPS figure. (Hivenet RTX 5090 benchmark)

The benchmark is more useful for that workload because it measures the thing the system was asked to do.

The most useful LLM performance metrics

For language-model serving, peak chip metrics quickly become secondary.

A small group of workload metrics tells you far more.

Time to first token

TTFT measures the delay between sending a request and receiving the first generated token.

Users feel this delay directly.

A model that eventually generates text extremely quickly can still feel slow if nothing happens for several seconds after the user presses Enter.

TTFT includes work such as request processing and prompt prefill.

For interactive applications, report percentiles rather than one average.

P50 tells you the typical experience.

P95 and P99 show what slower users experience.

Our guide to LLM inference metrics covers TTFT, throughput, queues, memory headroom, and cache behavior in more detail.

Time per output token

TPOT measures the average time required to generate each token after the first token arrives.

Lower TPOT usually means smoother generation.

The inverse is closely related to token generation rate.

For interactive text, both TTFT and TPOT matter.

One measures how long the user waits before anything happens.

The other measures how quickly the answer continues after generation begins.

MLCommons uses TTFT and time per output token in current LLM server benchmarks because the two phases represent different aspects of serving performance. (MLPerf LLM inference)

Tokens per second

Tokens per second measures token generation throughput.

Unfortunately, the metric is often reported ambiguously.

It can mean:

• tokens per second for one request;
• output tokens per second;
• total system tokens per second across all concurrent requests;
• input and output tokens combined.

Those numbers are not interchangeable.

A benchmark that says “2,000 tokens/s” without explaining what is being counted is difficult to interpret.

Always ask:

per request or total?

input or output?

at what concurrency?

under what latency limit?

Throughput

Throughput measures how much useful work a system completes over time.

For LLM serving, that might be:

• requests per second;
• output tokens per second;
• total tokens per second.

For vision:

• images per second;
• frames per second.

For an embedding service:

• documents per second;
• tokens embedded per second.

Throughput is particularly important for production economics because it determines how much demand one accelerator can absorb.

High throughput is less impressive if users wait unacceptably long for responses.

That is why latency and throughput need to be measured together.

Latency and throughput are often a trade-off

Batching several requests together can make an accelerator much more efficient.

It may also make individual requests wait.

Increase batching and you might improve total throughput while worsening latency.

Reduce batching and individual users may receive faster responses while accelerator utilization drops.

Neither outcome is universally better.

An offline document-processing job may prefer maximum throughput.

An autocomplete system may care much more about latency.

MLPerf explicitly uses different scenarios and latency constraints because AI systems are deployed under different workload patterns. The current MLPerf Inference suite measures complete systems using defined datasets, quality targets, request patterns, and performance metrics rather than one peak arithmetic specification. (MLPerf Inference)

That is a much healthier way to compare AI hardware.

Concurrency changes the answer again

A benchmark with one user may tell you almost nothing about a service expected to handle 1,000 users.

As concurrency rises:

• queues form;
• batches grow;
• KV-cache memory grows;
• GPU utilization changes;
• TTFT changes;
• throughput changes;
• memory can run out.

A GPU that appears slower in single-request testing may become much more efficient once the serving engine batches many requests.

Another accelerator may excel at low-latency single-stream generation but scale differently under concurrency.

The only useful benchmark is the traffic pattern that resembles your application.

P50 is not enough

Average latency is comforting.

Tail latency is where production problems hide.

Suppose a service has:

• P50 TTFT: 300 ms
• P95 TTFT: 2.4 s
• P99 TTFT: 8.1 s

The average may still look reasonable.

A meaningful portion of users is having a bad experience.

For interactive systems, measure at least:

• P50;
• P95;
• P99.

And test them while the service is under realistic load.

Model quality belongs in every performance benchmark

A model that produces answers twice as fast after aggressive quantization may also produce worse answers.

That is not necessarily a performance improvement.

It may be a different product.

Hardware and model benchmarks therefore need quality constraints.

MLPerf uses accuracy or quality requirements alongside performance tests so systems cannot claim higher speed merely by degrading the workload beyond the benchmark's accepted quality target. (MLPerf Inference documentation)

The same principle should apply to your own testing.

When comparing:

• FP16 with INT8;
• one quantization method with another;
• one accelerator with another;
• one serving engine with another;

check that the resulting model still performs the task well enough.

Speed without acceptable output quality has little operational value.

What should you measure for AI training?

Training has different goals from inference.

The useful metrics shift.

Time to train

How long does the model take to reach the required quality?

This is often the most direct training-performance metric.

A machine producing more raw FLOPS may still take longer if communication, memory, software, or scaling is poor.

MLPerf Training follows this logic by measuring the time required to train defined models to specified quality targets. (MLPerf Training)

Training throughput

Depending on the workload, throughput may be measured as:

• samples per second;
• images per second;
• tokens per second;
• sequences per second.

This tells you how much training data the system processes over time.

Again, the metric is meaningful only if the training configuration and quality target remain comparable.

Scaling efficiency

A second GPU does not automatically halve training time.

Neither do eight GPUs produce an automatic eightfold improvement.

Distributed workloads need processors to exchange gradients, activations, parameters, and other data.

Interconnect and communication efficiency therefore become important.

Hivenet's current GPU VM vs bare metal benchmark measures NCCL AllReduce bandwidth on an eight-RTX-5090 host because communication performance becomes relevant once work spans several GPUs.

This is another example of measuring the actual bottleneck rather than quoting peak chip throughput.

Our next article on training vs inference hardware will treat those differences in detail.

What should you measure for edge AI?

Edge systems have a different hierarchy of concerns.

A device may care about:

• inference latency;
• sustained performance;
• memory footprint;
• energy per inference;
• watts under continuous load;
• thermal throttling;
• model size;
• accuracy;
• offline reliability.

Peak TOPS can help you identify whether an NPU or embedded accelerator is in roughly the right class.

It will not tell you whether the device can run your model for eight hours inside a sealed enclosure without throttling.

That is why edge AI hardware needs to be evaluated as a complete physical system.

Energy efficiency needs a unit of useful work

Watts tell you power consumption.

They do not tell you efficiency.

Imagine:

• Processor A draws 100 W for ten seconds.
• Processor B draws 300 W for one second.

Processor B has a higher power draw.

Processor A used more total energy for that task.

A better comparison might therefore use:

• joules per inference;
• images per joule;
• tokens per joule;
• completed jobs per kilowatt-hour.

Even then, keep the output quality and latency requirement constant.

A low-power accelerator that cannot meet the service's throughput target is not necessarily the more efficient deployment.

Cost needs a unit of useful work too

Hourly GPU prices are easy to compare.

They can also be misleading.

Suppose:

• GPU A costs €1 per hour.
• GPU B costs €2 per hour.

If GPU B completes three times as many acceptable requests during that hour, its cost per request is lower.

Useful economic metrics include:

• cost per inference;
• cost per image;
• cost per million tokens;
• cost per training run;
• cost to reach a target model quality;
• cost per concurrent user under a latency target.

For specialized accelerators, also include engineering and migration costs.

Our AI accelerator vs GPU inference guide explains why a theoretically efficient inference chip may still be expensive if the software transition consumes weeks of engineering time.

Utilization determines how much performance you are actually buying

A GPU at 20% utilization can be a poor economic choice even if its benchmark numbers are excellent.

The goal is not to own the fastest processor.

It is to keep enough of the processor doing useful work.

Inference engines improve utilization through techniques such as:

• continuous batching;
• paged KV-cache management;
• optimized kernels;
• quantization;
• request scheduling.

Hivenet's existing LLM inference production guide and inference metrics guide cover these serving concerns directly.

This is an important reason to optimize software before replacing hardware.

Software can change hardware performance dramatically

Two identical GPUs can produce different benchmark results.

One may use:

• an older framework;
• inefficient kernels;
• a different CUDA version;
• poor batching;
• an unsuitable quantization format;
• unoptimized memory allocation.

The hardware specification has not changed.

The system performance has.

Specialized accelerators make the software relationship even more visible because models often need vendor-specific compilers, runtimes, or execution providers.

That is why our AI accelerator guide treats software support as part of the hardware decision.

For real AI systems:

hardware performance = hardware + model + precision + runtime + software configuration + workload

Remove any one term and the comparison becomes less useful.

NPU TOPS are useful within the right comparison

None of this means TOPS is a bad metric.

It becomes useful when you compare similar hardware under similar conditions.

For example, two NPUs from the same product generation that report dense INT8 peak performance using the same counting method can reasonably be compared by TOPS as one part of the decision.

If one offers 50 TOPS and another offers 20, the first has more theoretical AI arithmetic capability under those conditions.

Then ask the next questions:

• Does my model run on both?
• Do they support the same operators?
• Which memory system does each use?
• What power do they consume?
• What performance do they sustain?
• How fast is my actual model?

TOPS helps narrow the field.

It should not finish the evaluation.

For more on where NPUs fit beside other processors, see what an NPU is and CPU vs GPU vs NPU.

FLOPS are useful within the right comparison too

The same principle applies to FLOPS.

Comparing FP32 throughput between similar GPU architectures can tell you something useful about their theoretical ability to execute FP32 arithmetic.

Comparing BF16 or FP8 tensor throughput can help when the workload actually uses those formats.

Problems arise when the precision, execution path, sparsity assumption, or workload changes between the numbers being compared.

A benchmark table should therefore never contain a bare column called simply:

Performance

Use:

FP32 TFLOPS

INT8 TOPS

BF16 tensor throughput

or whatever the number genuinely represents.

Specificity makes the comparison much harder to misuse.

Peak specs are useful for architecture planning

Peak specifications still have an important role.

This is especially important when comparing AI accelerator cards with different chips or FPGA and GPU implementations that use different precisions and workload assumptions.

They can help answer questions such as:

• Is this hardware remotely large enough for the job?
• Does this accelerator support the precision I need?
• How much memory is available?
• Which generation contains the required execution units?
• Is the device designed for my power envelope?
• Which systems are worth benchmarking?

The mistake is skipping the benchmark and treating the peak number as the answer.

A better AI performance scorecard

When evaluating hardware, build a scorecard around the workload.

For LLM inference

Measure:

‍

‍

For training

Measure:

‍

‍

For edge AI

Measure:

‍

‍

That tells you much more than TOPS alone.

How to compare two AI processors properly

A defensible comparison needs controls.

Use the same model

Model architecture has a huge effect on performance.

Do not compare one processor running Llama 8B against another running a smaller model and call it a hardware benchmark.

Use the same model quality

If one version is quantized more aggressively, validate that quality remains acceptable.

Record precision

State FP16, BF16, FP8, INT8, INT4, FP4, or whatever the workload actually uses.

Record model size

Include the number of parameters and relevant architecture details.

Record input size

For LLMs, prompt length can change performance dramatically.

Record output size

Generating 32 tokens and 2,000 tokens are different workloads.

Record batch size

Batching changes utilization and latency.

Record concurrency

One request is not a production service.

Record software

Include:

• framework;
• serving engine;
• drivers;
• compiler;
• accelerator runtime;
• relevant library versions.

Record memory

List available memory and peak workload use.

Record latency and throughput together

Optimizing one can damage the other.

Record quality

A faster model that no longer performs the task is not an equivalent result.

Record power if efficiency matters

Explain how it was measured.

Record price if you make a cost claim

Hardware and cloud pricing change.

Date the comparison.

This is the approach Hivenet now uses on its benchmark pages, where model, precision, GPU configuration, request pattern, latency, and other test conditions are documented alongside results.

Why MLPerf is more useful than a TOPS comparison

MLCommons exists partly because hardware comparisons need shared rules.

Its benchmark suites define:

• models;
• datasets;
• quality targets;
• request patterns;
• latency requirements;
• measurement procedures.

MLPerf Client now benchmarks LLM workloads across GPUs, NPUs, and hybrid client configurations rather than assuming that advertised NPU TOPS predicts useful local AI performance. (MLPerf Client)

MLPerf Inference does the same thing for servers and other deployment classes.

This makes comparisons less tidy than reading specification sheets.

It also makes them much more useful.

Why 3,352 AI TOPS does not make an RTX 5090 67 times faster than a 50-TOPS NPU

Return to the comparison at the beginning.

NVIDIA advertises the RTX 5090 at 3,352 AI TOPS.

Several current client NPUs sit around 40 to 50 TOPS.

The temptation is to divide.

Resist it.

You would be comparing:

• different processor classes;
• different power envelopes;
• different memory systems;
• potentially different precision and counting assumptions;
• different software environments;
• different target workloads.

The RTX 5090 has 32 GB of dedicated GDDR7 and enormous memory bandwidth.

A laptop NPU usually works inside a shared-memory client system designed around a much lower power budget.

The NPU may be the sensible processor for continuous video-call effects.

The RTX 5090 may be the sensible processor for high-throughput LLM serving.

One is not “67 times better.”

They are designed around different constraints.

That is precisely why our NPU vs GPU guide starts with the workload rather than peak arithmetic.

What is a good TOPS number for AI?

There is no universal target.

A good TOPS figure is one attached to hardware that meets the workload's requirements.

For a Windows Copilot+ PC, Microsoft's current 40+ NPU TOPS requirement gives the number a concrete platform meaning.

For a tiny vision model, much less compute may be sufficient.

For a large server workload, thousands of TOPS may still tell you less than:

• available memory;
• actual latency;
• aggregate throughput;
• supported precision;
• software stack;
• cost.

Instead of asking:

How many TOPS do I need?

ask:

What model do I need to run, under what latency, memory, power, and cost constraints?

Then benchmark suitable hardware.

What AI performance metric matters most?

It depends on the workload.

‍

‍

TOPS and FLOPS appear near the beginning of several of those decisions.

They rarely appear at the end.

Where Hivenet performance numbers fit

Hivenet publishes hardware specifications because they help users understand available Compute capacity.

But the more useful numbers are the workload measurements.

The current Hivenet benchmark library includes GPU virtualization, multi-GPU communication, AI inference, OCR, and API testing.

For LLM inference, Hivenet measures variables such as:

• latency;
• token throughput;
• concurrency;
• model;
• precision;
• GPU configuration;
• request pattern.

That is how you should evaluate hardware you plan to rent through Compute with Hivenet, too.

If the workload is small enough for CPU compute, use CPU compute.

If an NPU already inside the end device handles it well, use the NPU.

If a larger or changing workload benefits from GPU infrastructure, benchmark it on a GPU.

If you only need a managed model endpoint, the Hivenet Inference API moves the decision up another level, where endpoint latency, throughput, model quality, region, and cost become more important than the processor specification.

The metric should follow the thing you are actually buying.

FAQ about TOPS, FLOPS, and AI performance

What does TOPS stand for in AI?

TOPS stands for trillions of operations per second. It describes a processor's theoretical ability to perform trillions of numerical operations each second under specified conditions.

What does 40 TOPS mean?

Forty TOPS means a theoretical peak of 40 trillion operations per second under the conditions used to calculate the specification. Microsoft currently requires 40+ NPU TOPS for many Copilot+ PC AI features, which is why the number appears frequently in AI PC marketing.

Is 50 TOPS good for AI?

It can be plenty for supported local AI workloads. The number alone cannot tell you whether a particular model will run well. Check memory, precision, software support, latency, sustained performance, and model compatibility.

Is more TOPS always better?

Higher TOPS means greater theoretical arithmetic capability when the figures are calculated on comparable terms. It does not guarantee proportionally better application performance.

What is the difference between TOPS and TFLOPS?

TOPS counts operations more generally and is often used for low-precision AI acceleration. TFLOPS counts trillions of floating-point operations per second. Neither metric is meaningful without knowing the precision and execution conditions.

Can I compare GPU TOPS with NPU TOPS?

Only with considerable care. GPUs and NPUs can use different architectures, numerical formats, counting assumptions, memory systems, and software. Running the same benchmark workload on both is much more useful than comparing the peak specifications directly.

Does TOPS measure LLM performance?

No. For LLM inference, measure metrics such as time to first token, time per output token, tokens per second, concurrency, memory use, and cost. TOPS may describe one hardware capability involved in producing those results.

Does FLOPS measure AI training performance?

FLOPS describes floating-point computational throughput and can help characterize training hardware. Actual training performance also depends on memory, communication, software, model architecture, utilization, and scaling. Time to reach a defined quality target is often more useful.

What is the difference between FLOP and FLOPS?

A FLOP can mean one floating-point operation or, in model analysis, an amount of computational work. FLOPS means floating-point operations per second and describes a processing rate.

Why do lower-precision formats produce higher AI performance numbers?

Smaller numerical formats require fewer bits, allowing hardware to store, move, and often process more values with the same physical resources. Whether that translates into faster application performance depends on the available hardware and software kernels.

Why is memory bandwidth important for AI?

Processors need a continuous supply of model weights, activations, and other data. If computation waits for memory, additional arithmetic capability may remain unused. Memory-bound workloads such as parts of LLM decoding can therefore depend heavily on bandwidth.

Which metric should I use to compare LLM GPUs?

Start with the same model and precision, then compare TTFT, TPOT, single-request and aggregate token throughput, concurrency, memory use, and cost under the latency target you expect in production.

What is the best AI benchmark?

There is no single benchmark for every AI workload. MLPerf provides standardized benchmark suites across training, inference, client systems, and other deployment classes. Your final hardware decision should still include a benchmark using your own model and traffic pattern.

Measure the work, not the label on the box

TOPS and FLOPS solve a useful problem.

They compress complicated processor architectures into numbers that help describe theoretical computational capacity.

The trouble starts when the number becomes a substitute for testing.

AI applications are complete systems.

They contain models, memory, software, numerical formats, queues, users, networks, storage, power limits, and cost constraints.

A processor can have enormous theoretical throughput and still be the wrong hardware.

A modest accelerator can be exactly right if it executes the required model within the latency and power budget.

For LLM inference, users experience TTFT and token generation speed.

Infrastructure teams experience throughput, memory pressure, queues, utilization, and cost.

Training teams experience time to convergence and scaling.

Edge systems experience latency, heat, energy, and physical constraints.

Those are the measurements that decide whether the hardware is doing its job.

Use TOPS and FLOPS to understand what a processor might be capable of.

Use a workload benchmark to find out what it can actually do.

Continue with NPU vs GPU for AI workloads, what an NPU is, the practical guide to AI accelerators, CPU vs GPU vs NPU, AI accelerators vs GPUs for inference, edge AI hardware, and training vs inference hardware.

‍