Metrics that matter for LLM inference

A benchmark says:

5,000 tokens per second

Is that good?

There is no way to know yet.

Five thousand tokens per second from which model? On how many GPUs? At what precision? With what prompt length? How many simultaneous requests? How long did users wait for their first token? Were those output tokens or input and output tokens combined? What happened to P95 latency? Did every request finish?

LLM inference benchmarks are unusually easy to make impressive and unusually easy to misread.

That is because an inference server is trying to optimize several things at once:

• response latency
• generation speed
• total throughput
• concurrency
• GPU memory
• model quality
• cost

Improving one can make another worse.

A useful benchmark therefore does more than answer:

How fast did the GPU go?

It answers:

How much useful work did this complete, at what latency, under what load, using what model and hardware?

That is the question to keep in mind throughout this guide.

One LLM request has several different stages

Before looking at benchmark metrics, it helps to see what happens during one request.

A simplified request looks like this:

request sent
    ↓
network
    ↓
queue
    ↓
prefill
    ↓
first output token
    ↓
decode
    ↓
token → token → token → token
    ↓
final token

The prefill phase processes the input prompt.

The decode phase generates output autoregressively, normally one token at a time.

Those phases behave differently. Prefill processes many input tokens and is often relatively compute-intensive. Decode repeatedly loads model state while producing individual output tokens and can become heavily constrained by memory bandwidth and serving conditions.

The metrics used in LLM inference benchmarking are different views of this timeline. NVIDIA's current benchmarking guidance identifies TTFT, end-to-end latency, inter-token latency, tokens per second, and requests per second among the central measurements for LLM serving.

Here is the useful map:

You rarely want only one of them.

‍

‍

TTFT measures the wait before the answer starts

Time to first token, or TTFT, measures the interval from sending the request until the first actual generated token arrives.

Conceptually:

request sent
|--------------------------------|
                            first token

<------------- TTFT ------------->

TTFT can include:

• network time
• queueing
• tokenization
• prompt processing
• scheduling
• generation of the first output token

The exact boundary depends on where the benchmark client measures from, which is one reason two benchmarking tools should not be assumed to calculate every metric identically. NVIDIA explicitly warns that popular benchmarking tools can differ in how apparently similar LLM metrics are measured.

For a streaming chatbot, TTFT is highly visible.

A response can generate extremely quickly once it begins and still feel slow if the interface spends several seconds doing nothing first.

Long prompts usually put more pressure on TTFT

Before generating the first output token, the model has to process the prompt.

That is prefill.

A request containing:

500 input tokens

and one containing:

50,000 input tokens

therefore present very different workloads before decoding starts.

This is why comparing TTFT across benchmarks with different input lengths is usually meaningless.

NVIDIA's benchmark methodology emphasizes input and output sequence-length distributions because prompt length changes prefill work, memory requirements, and latency characteristics.

If one provider benchmarks 128-token prompts and another uses 16K-token prompts, their TTFT numbers are not measuring equivalent jobs.

TPOT measures generation after the first token

Time per output token, or TPOT, describes the average time required to produce output tokens after generation has started.

Current vllm bench serve calculates per-request TPOT roughly as:

end-to-end latency - TTFT
─────────────────────────
   output tokens - 1

The first token is excluded because its latency belongs to TTFT rather than steady-state decoding.

If TPOT is:

10 ms

the corresponding steady-state generation rate is roughly:

1 / 0.010
=
100 tokens/s

for that request.

Lower TPOT is better.

Unlike TTFT, TPOT is mainly describing what happens after the answer has already begun.

ITL measures what streaming actually feels like

Inter-token latency, or ITL, measures the delay between consecutive streamed output tokens.

Imagine:

token      token         token   token
 ↓          ↓             ↓       ↓
 ●----8ms---●-----21ms-----●--7ms--●

TPOT gives you an average decoding rate.

ITL lets you see the individual gaps.

Current vLLM benchmarking reports both separately. Its implementation calculates TPOT from total post-first-token generation time, while ITL is collected from the actual intervals between streamed output events.

That distinction can expose jitter.

Two requests might have similar average TPOT while one produces:

10ms
10ms
11ms
9ms
10ms

and the other:

2ms
3ms
37ms
2ms
6ms

The averages can look acceptable while the second stream feels much less consistent.

Some benchmark literature and tools use TPOT and average ITL almost interchangeably. Others distinguish them. When comparing published results, check the tool's definition rather than assuming that identical acronyms mean identical measurement methods.

End-to-end latency measures the complete request

End-to-end latency measures the time from request submission until the complete response has been received.

For a simple streamed request:

E2E latency
=
TTFT
+
generation time

NVIDIA uses that same basic relationship in its inference benchmarking definitions.

End-to-end latency becomes especially important for workloads where users need the finished answer rather than the first token.

Examples include:

• classification
• extraction
• structured JSON
• summarization used by another service
• agent steps
• background processing

For a chat interface, TTFT and ITL may describe perceived responsiveness better.

For an automated pipeline waiting for a complete JSON object, end-to-end latency may be the number that matters most.

Tokens per second is where benchmarks become dangerous

“Tokens per second” sounds wonderfully precise.

The problem is that people use it for several different things.

A benchmark might mean:

Per-user output speed

How quickly one response generates.

Request A → 90 tokens/s

Total output throughput

How many output tokens the entire inference server generates across all active requests.

Server → 8,000 output tokens/s

Total token throughput

Some tools also report a number containing both:

input tokens
+
output tokens

These numbers answer different questions.

NVIDIA's inference benchmarking tools explicitly distinguish total output throughput from per-user output throughput and total token throughput.

So when somebody says:

This server does 10,000 TPS.

your next question should be:

Which TPS?

Per-user speed and system throughput can move in opposite directions

Suppose one request runs alone.

It might generate at:

150 tokens/s per user

Now increase concurrency.

The GPU can batch work from many requests together, becoming more efficient overall.

Perhaps the server reaches:

8,000 total tokens/s

but an individual request now receives:

80 tokens/s

Neither number is wrong.

The server improved its aggregate throughput while individual generation became slower.

That tradeoff is fundamental to LLM serving.

A provider quoting only aggregate TPS can therefore make a heavily batched system look spectacular while hiding what one user experiences.

A provider quoting only single-request TPS can make a server look responsive while telling you nothing about how much traffic it can handle.

You need both perspectives.

Requests per second measures completed jobs

Requests per second, or RPS, tells you how many requests the system completes over time.

This is particularly useful when requests have relatively consistent shapes.

For example:

12.5 completed requests/s

can be meaningful for a classification endpoint where every request has roughly:

500 input tokens
20 output tokens

But RPS becomes harder to compare when one benchmark contains tiny requests and another contains huge ones.

One request asking for a one-word classification and one request asking for a 4,000-token report both count as:

1 request

Token counts and request shapes therefore need to sit beside RPS.

Goodput asks whether the completed work was actually good enough

Throughput counts work that finished.

Goodput counts work that finished while satisfying the service's performance requirements.

Suppose a server completes:

100 requests/s

but your service requirement is:

TTFT < 1 second

and only:

72 requests/s

meet it.

Then:

throughput = 100 req/s
goodput    = 72 req/s

For the user-facing service, 72 may be the more meaningful capacity number.

Current vllm bench serve can calculate goodput against explicit TTFT, TPOT, and end-to-end latency service-level objectives.

That makes goodput one of the most useful concepts in production benchmarking.

It connects raw hardware performance to the experience the application actually promises.

Percentiles matter more than averages

Suppose a benchmark reports:

average TTFT = 450 ms

That looks good.

But imagine the distribution is:

P50 = 220 ms
P95 = 1.8 s
P99 = 6.4 s

A significant part of the traffic is having a very different experience from the average.

That is why useful benchmark reports include percentiles such as:

• P50
• P90
• P95
• P99

Current vLLM benchmarking can report percentiles independently for TTFT, TPOT, ITL, and end-to-end latency.

The higher percentiles describe tail latency.

Tail latency matters because production systems are not judged exclusively by their median user.

If an agent consists of ten sequential model calls, an occasional very slow request can also compound through the workflow.

P50, P95, and P99 answer different questions

A useful mental model is:

P50
What does a typical request experience?

P95
What does a relatively unlucky but still common request experience?

P99
How bad does the service get near the tail?

For an internal batch workload, average throughput may dominate.

For an interactive product, P95 TTFT may be far more useful.

For an API with a strict deadline, P99 end-to-end latency might determine whether the service is acceptable.

There is no universally correct percentile.

There is a percentile that corresponds to the product requirement.

Benchmark results are meaningless without the workload

This is the single most important rule when comparing LLM inference systems.

You need to know what work was actually performed.

At minimum, a credible benchmark should identify:

Hivenet's own benchmark methodology follows the same principle: published results expose workload, hardware, environment, model or data, load profile, metrics, baseline, result, limitations, and test date.

‍

‍

Without the test conditions, a benchmark result is difficult to reproduce and dangerous to compare.

Input and output length deserve particular attention

Imagine two RTX 5090 benchmarks.

Benchmark A

input = 128 tokens
output = 128 tokens

Benchmark B

input = 16,384 tokens
output = 2,048 tokens

Even with the same model and serving engine, these are radically different workloads.

Benchmark B has:

• much more prefill work
• more KV-cache use
• substantially longer decode
• different batching behavior
• different opportunities for request overlap

If A produces more TPS, that tells you almost nothing about which deployment is better for workload B.

Real prompt and response distributions belong in the benchmark.

Context limit is not the same thing as prompt length

A server might be configured to support:

128K context

while the benchmark sends only:

1K prompts

That is useful information.

But it does not demonstrate performance at 128K.

The context limit tells you what the server permits.

The benchmark request shape tells you what it tested.

Our KV cache guide explains why long active context consumes increasing GPU memory and can change serving capacity dramatically.

If long-context performance matters, test long contexts.

Concurrency is where the real serving curve appears

A benchmark at concurrency 1 tells you something useful:

the low-load latency floor.

It does not tell you how the system behaves as traffic increases.

A proper serving benchmark should normally sweep concurrency or request rate.

At low concurrency:

low latency
low aggregate throughput

As concurrency rises:

batching improves
GPU utilization rises
aggregate throughput rises

Eventually:

throughput approaches saturation
queueing increases
latency rises sharply

NVIDIA's current inference-capacity methodology explicitly uses latency-throughput curves across concurrency levels for this reason.

Conceptually:

Throughput
   ^
   |                       _________
   |                   ___/
   |                __/
   |             __/
   |___________/
   +-----------------------------> concurrency


Latency
   ^
   |                         /
   |                       _/
   |                    __/
   |___________________/
   +-----------------------------> concurrency

The interesting part is not necessarily the highest point on the throughput curve.

It is the best operating point before latency becomes unacceptable.

Saturation is where more traffic stops buying useful throughput

Suppose you measure:

Moving from:

1 → 16 concurrent requests

buys a large increase in throughput with a moderate latency increase.

Moving from:

32 → 64

adds very little throughput while P95 TTFT becomes dramatically worse.

For an interactive service, concurrency 64 is probably not the winning configuration even though it produces the largest TPS number.

This is why:

maximum benchmark throughput

and:

useful production capacity

are different things.

The best result usually sits on a latency-throughput tradeoff

Infrastructure benchmarking is a multi-objective problem.

You may want:

lower TTFT
higher throughput
lower cost
more concurrency
more model quality

at the same time.

Usually, you cannot maximize all five.

NVIDIA describes this with a Pareto frontier: a deployment is attractive when no alternative provides higher throughput at the same or lower latency.

You do not need formal optimization mathematics to use the idea.

If configuration B is:

slower
more expensive
and
lower throughput

than configuration A under the same workload, B is not an interesting deployment.

If B costs more but substantially improves latency, now you have a real tradeoff to evaluate.

Quantization should be benchmarked as a quality-performance trade

Suppose you compare:

BF16
FP8
NVFP4

Quantization may improve:

• weight memory
• GPU bandwidth pressure
• throughput
• GPU count
• KV-cache headroom

But a benchmark containing only latency and TPS is incomplete.

You also need to establish that the quantized model still performs the task adequately.

Our NVFP4 guide explains why model quality needs to be evaluated alongside memory and throughput.

A 4-bit model that generates 30% faster but breaks your structured output is not a 30% improvement.

Inference benchmarking and model-quality evaluation are separate tests.

You need both.

Tensor parallelism needs cost-per-result benchmarking

Suppose the same model can run at:

TP = 2

or:

TP = 4

Four GPUs may reduce latency.

They may also double the hardware used by the replica.

The comparison should therefore include:

• TTFT
• TPOT / ITL
• aggregate throughput
• throughput per GPU
• cache capacity
• concurrency
• cost

Our tensor parallelism guide explains why adding GPUs does not produce linear scaling: communication and synchronization become part of every distributed forward pass.

The benchmark tells you whether the additional GPUs actually earn their place.

PagedAttention and batching mainly show up under load

Running one request does not tell you much about a serving scheduler.

Techniques such as:

• PagedAttention
• continuous batching
• chunked prefill
• prefix caching

become important as requests overlap and compete for GPU memory and compute.

That is why our PagedAttention and continuous batching guide focuses on concurrent serving rather than single-request speed.

If you are testing scheduling efficiency, use a workload that creates a scheduling problem.

A good benchmark also reports errors

Imagine:

10,000 requests attempted
9,300 succeeded
700 failed with OOM

If the benchmark reports throughput from only the successful requests and never mentions the failures, the number is misleading.

Useful stability signals include:

• completed requests
• timeouts
• OOMs
• HTTP errors
• cancellations
• request preemptions
• retry rate

Hivenet's benchmark methodology explicitly includes error rate where it is relevant because throughput is only useful while the setup remains stable.

A system that produces excellent throughput immediately before falling over is not high capacity.

Warm-up matters

The first few requests after loading a model can behave differently from steady-state traffic.

Potential causes include:

• kernel compilation
• CUDA graph setup
• memory allocation
• cache warming
• disk/model loading
• runtime initialization

If one benchmark includes cold-start requests and another measures an already warm server, their latency numbers may differ for reasons unrelated to normal serving performance.

State whether the benchmark is measuring:

cold start

or:

steady state

and how warm-up requests were handled.

Synthetic benchmarks are useful when you know what they represent

Synthetic requests make controlled comparison easy.

You can hold constant:

input length
output length
request rate
concurrency

and change one system variable.

That is excellent for A/B testing hardware or serving configuration.

It is less useful if your production traffic has a very different shape.

Current vllm bench serve supports synthetic and custom datasets and can output distributions for prompt, output, and combined token counts.

Use synthetic traffic to isolate variables.

Use representative traffic to predict production.

Those are different benchmark goals.

Load testing and benchmarking are related but different

A performance benchmark asks:

How does this model-serving configuration perform under defined conditions?

A load test asks:

What happens when we put realistic or extreme traffic pressure on the service?

NVIDIA draws a similar distinction, describing load testing as a way to evaluate capacity, scaling, network behavior, and resource utilization under traffic, while performance benchmarking focuses more directly on model-serving throughput and latency.

You normally want both before production.

The benchmark helps choose the configuration.

The load test helps discover where it breaks.

How to read a benchmark result in practice

Imagine this report:

Model:             Example-27B
Precision:         NVFP4
GPU:               1 × RTX 5090
Input length:      2,048 tokens
Output length:     512 tokens
Concurrency:       16

Output throughput: 2,400 tokens/s
Request throughput: 4.7 req/s

TTFT
P50: 260 ms
P95: 610 ms
P99: 940 ms

TPOT
P50: 8.2 ms
P95: 11.4 ms

Errors: 0

Do not begin with:

2,400 tokens/s

Begin with the test conditions.

One RTX 5090. A 27B model. NVFP4. 2K prompts. 512-token outputs. Concurrency 16.

Now the performance numbers have meaning.

For an interactive application, you might conclude:

• typical first-token latency is good
• P95 TTFT remains under one second
• steady-state generation is fast
• no errors occurred
• concurrency 16 achieves 2,400 output TPS

Then compare the same workload at concurrency 8 and 32.

That tells you whether 16 is an efficient operating point or merely one arbitrary measurement.

Now imagine a competing benchmark

Provider B says:

Output throughput: 3,200 tokens/s

It appears 33% faster.

But then you discover:

Input length:       128 tokens
Output length:      128 tokens
Concurrency:        64
P95 TTFT:           5.2 seconds

These are not directly comparable results.

Provider B ran:

• much shorter requests
• much more concurrency
• a dramatically worse tail TTFT

It may still be an excellent configuration for offline batch generation.

It may be a poor configuration for your interactive application.

The word faster cannot resolve that.

The workload and SLO can.

Benchmark red flags

I would be cautious with any LLM inference benchmark that does one or more of these.

Reports TPS without defining it

Is it:

per-user output TPS
total output TPS
total input + output TPS

Those are different metrics.

Gives no prompt or output lengths

Then you do not know what workload produced the result.

Gives no concurrency or request rate

Then you cannot tell whether it measures single-request latency or heavily batched throughput.

Reports only averages

Tail latency can disappear inside a good mean.

Compares different quantizations without evaluating quality

Faster output is not equivalent output if the model became materially worse.

Compares different model revisions

The hardware comparison is now contaminated by a model change.

Changes the serving engine and hardware simultaneously

You know the result changed.

You do not know which variable caused it.

Reports aggregate throughput without individual latency

Heavy batching can inflate the first while degrading the second.

Uses extremely short synthetic prompts for a long-context application

The benchmark does not represent the workload.

Ignores failures

A request that OOMs or times out did not become free merely because it is absent from the TPS calculation.

Benchmarks one concurrency level

You cannot see the latency-throughput curve or the saturation point.

Change one variable at a time when you want an explanation

Suppose you want to know whether FP8 KV cache helps.

Keep constant:

model
weights
GPU
GPU count
serving engine
input lengths
output lengths
request rate
concurrency
context limit

Change:

KV cache precision

Now you have an experiment.

If you simultaneously move from:

BF16 → NVFP4
1 GPU → 2 GPUs
vLLM → TensorRT-LLM
8K → 32K context

and throughput rises, you have demonstrated a different configuration.

You have not learned which optimization mattered.

Both kinds of benchmarks are useful.

Do not confuse them.

How to benchmark a vLLM endpoint

Current vLLM includes:

vllm bench serve

for online-serving benchmarks.

A representative command might look like:

vllm bench serve \
 --backend vllm \
 --model <model> \
 --base-url <endpoint> \
 --dataset-name random \
 --random-input-len 2048 \
 --random-output-len 512 \
 --num-prompts 1000

The exact CLI options evolve, so use the current vLLM documentation when running the test rather than treating this example as a pinned production command. Current versions can report TTFT, TPOT, ITL, E2E latency, output throughput, request throughput, percentile distributions, and goodput against latency SLOs.

The important part is not the command.

It is controlling the workload.

Run a concurrency or request-rate sweep

Do not stop at one test.

For example:

concurrency 1
concurrency 2
concurrency 4
concurrency 8
concurrency 16
concurrency 32
concurrency 64

For each level, record:

TTFT P50 / P95 / P99
TPOT P50 / P95 / P99
ITL
output TPS
RPS
errors

Then plot:

throughput vs TTFT

or:

throughput vs TPOT

The curve will tell you far more than one headline number.

This is also how Hivenet approaches foundational model-serving benchmarks: realistic prompt shapes, concurrency sweeps, latency metrics, end-to-end latency, and throughput rather than an isolated speed claim.

Benchmark your real prompt distribution too

After the controlled sweep, test something resembling production.

If your application normally receives:

60% → 1–2K prompts
30% → 4–8K prompts
10% → 20K+ prompts

use a dataset that reflects it.

Do the same for output length.

A server optimized for:

short prompt
long output

may behave differently from one serving:

long prompt
short output

even when their average total token counts look similar.

The shape matters because prefill and decode stress the system differently.

Benchmark caching only with workloads that can use it

If you are testing prefix caching, create repeated prefixes.

If you benchmark only random unrelated prompts, the system has little opportunity to reuse prefix state.

Likewise, if you want to evaluate KV-cache pressure, use enough context and concurrency to create meaningful cache demand.

A benchmark should create the problem the optimization claims to solve.

Otherwise the result tells you almost nothing about that feature.

Convert benchmark performance into cost

Once you have a stable throughput result, infrastructure economics become easier to calculate.

Suppose the deployment costs:

C currency units per hour

and produces:

T usable output tokens per second

Then:

output tokens per hour
=
T × 3,600

and a simple infrastructure cost per million output tokens becomes:

C
──────────────── × 1,000,000
T × 3,600

But remember the word usable.

If the configuration misses your latency SLO or produces unacceptable model quality, its cheap tokens are not equivalent to the tokens from a configuration that meets the product requirement.

This is why NVIDIA's capacity-planning methodology combines throughput with latency or quality-of-service constraints before turning benchmark performance into infrastructure sizing and cost.

Goodput can produce a better cost number

Suppose:

System A
10,000 output tokens/s
60% of requests meet SLO

and:

System B
8,500 output tokens/s
98% of requests meet SLO

Raw throughput favors A.

The production service may get more useful capacity from B.

You can apply the same logic to cost:

hourly infrastructure cost
──────────────────────────
work completed within SLO

That is a much more demanding metric.

It is also closer to what the business is paying for.

A benchmark should say what it does not prove

This is a good habit in technical writing.

A result such as:

Configuration A delivered lower P95 TTFT than configuration B at concurrency 16.

does not prove:

Configuration A is the better inference system for every workload.

Perhaps B performs better with long prompts.

Perhaps B scales further at high concurrency.

Perhaps A uses a lower-quality quantization.

Perhaps B costs half as much.

Hivenet's benchmark methodology explicitly includes a limits field for this reason: a result should state what the test supports and what it cannot establish.

That makes a benchmark more trustworthy, not less impressive.

Which metrics should you care about?

The answer depends on the workload.

Interactive chat

Prioritize:

TTFT
ITL / TPOT
P95 / P99 latency
goodput

The user needs the response to begin quickly and stream smoothly.

Agents

Watch:

TTFT
E2E latency
tail latency
error rate

One agent action may depend on several sequential model calls, so slow tails can compound.

RAG

Watch:

TTFT
prompt length
prefill performance
KV-cache use
throughput under mixed contexts

Retrieved context can make prompts much larger.

Coding

Watch:

TTFT
long-context behavior
generation speed
tail latency

Large repositories can create substantial prefill workloads.

Offline batch inference

Prioritize:

aggregate throughput
cost per useful token/request
error rate

Individual TTFT may matter very little.

Structured extraction

Prioritize:

E2E latency
RPS
correct output rate
error rate

A fast malformed JSON object is not a successful result.

There is no universally good TTFT or TPS

Questions such as:

What is a good TTFT?

or:

How many tokens per second should an LLM produce?

do not have one useful universal answer.

A:

2-second TTFT

may be terrible for autocomplete and entirely acceptable for a background research job.

A:

30 token/s

stream may feel perfectly readable to one user while being far too slow for an automated code-generation pipeline.

Define the product requirement first.

Then evaluate the benchmark against it.

Benchmark metrics are measurements.

The SLO gives them meaning.

The benchmark should help you make an infrastructure decision

A good benchmark ends with an action.

Perhaps:

NVFP4 lets the model fit on one GPU
without unacceptable quality loss.

Or:

TP=4 improves latency,
but TP=2 has much better throughput per euro.

Or:

Concurrency 32 is the highest point
that keeps P95 TTFT under our 1-second SLO.

Or:

FP8 KV cache lets us support
twice as many concurrent long-context requests.

Or:

The bigger GPU does not improve this workload
enough to justify its price.

That is the purpose of benchmarking.

Not producing the largest number.

Producing enough evidence to make a better deployment decision.

How Hivenet approaches benchmark results

Hivenet publishes GPU, inference, VM, and AI benchmark results with the test conditions attached.

For inference work, that means looking at metrics such as:

• TTFT
• TPOT
• ITL
• end-to-end latency
• throughput
• concurrency
• error rate

alongside the actual model, precision, hardware, software environment, workload, and load profile.

That is also the standard we should use when comparing Hivenet with another infrastructure option.

If the workload changes, say so.

If the benchmark tests only one part of the system, say so.

If a difference sits inside normal variance, say so.

The goal of a benchmark is to make the result easier to question and reproduce, not harder.

The useful way to read any LLM benchmark

When someone shows you an inference benchmark, read it in this order.

1. What model was tested?

Check model revision, precision, and quantization.

2. What hardware was used?

Check GPU type and GPU count.

3. What did each request look like?

Check input and output token distributions.

4. How much load was applied?

Check concurrency and request rate.

5. What did users experience?

Look at TTFT, TPOT/ITL, E2E latency, and tail percentiles.

6. How much work did the server complete?

Look at output TPS and RPS.

7. Did it stay stable?

Check errors, OOMs, timeouts, and preemptions.

8. Did the model remain good enough?

Performance benchmarking does not replace quality evaluation.

9. What did it cost?

Compare useful throughput or goodput against the hardware cost.

10. What does the test not prove?

Look for the boundaries of the result.

If those ten answers are available, you probably have a benchmark worth using.

If all you have is:

12,438 TOKENS/SECOND!!!

you mostly have a number.

LLM inference benchmark FAQ

What is an LLM inference benchmark?

An LLM inference benchmark measures the performance of a deployed language model under defined workload conditions. Useful benchmarks normally measure latency and throughput while documenting the model, hardware, precision, request shape, and load.

What is TTFT?

TTFT means time to first token. It measures how long the user waits from submitting a request until the first generated token arrives.

What affects TTFT?

TTFT can be affected by network latency, queueing, scheduling, prompt length, prefill performance, model size, hardware, batching, and current server load.

What is TPOT?

TPOT means time per output token. It describes average steady-state generation time after the first output token. Current vLLM calculates it from post-TTFT generation time divided by the number of remaining output tokens.

What is ITL?

ITL means inter-token latency, the delay between consecutive streamed output tokens. It helps reveal whether generation is smooth or contains large latency spikes.

Are ITL and TPOT the same?

They describe closely related decode performance, and some tools use the terms similarly. Current vLLM reports both: TPOT is a per-request average over post-first-token generation, while ITL records the individual gaps between output events.

What is end-to-end latency?

End-to-end latency measures the time from submitting the request until the complete response is received. It includes the initial wait and the generation period.

What does tokens per second mean for an LLM?

It can mean different things. A benchmark may report individual generation speed, total server output throughput, or total input-plus-output throughput. Always check the metric definition before comparing TPS results.

Is higher TPS always better?

No. Aggregate throughput can increase as concurrency rises while individual-request latency becomes worse. Evaluate throughput together with TTFT, TPOT/ITL, and your latency targets.

What is RPS?

RPS means requests per second. It measures how many requests the system completes each second.

What is goodput?

Goodput measures how much work completes while satisfying defined performance requirements. Current vLLM can calculate request goodput against TTFT, TPOT, and end-to-end latency SLOs.

Why should an LLM benchmark report P95 or P99?

Averages can hide slow requests. Percentiles show tail latency and help determine what relatively unlucky users experience under load.

What is a concurrency sweep?

A concurrency sweep repeats the same benchmark at increasing numbers of simultaneous requests. It reveals how throughput and latency change as the system approaches saturation.

Why does prompt length matter in an LLM benchmark?

Longer prompts require more prefill computation and normally consume more inference memory. Comparing systems with different prompt lengths can therefore produce misleading latency and throughput conclusions.

Why does output length matter?

Longer outputs create more autoregressive decode work and keep requests active for longer. They also increase KV-cache state as generation proceeds.

What is the difference between an LLM benchmark and a load test?

A benchmark measures performance under controlled conditions. A load test focuses on how the service behaves as realistic or extreme traffic stresses capacity, scaling, queues, networking, and stability.

What is the best metric for an interactive chatbot?

TTFT and ITL/TPOT are particularly useful for perceived responsiveness, but throughput, P95/P99 latency, and goodput matter once multiple users share the service.

What is the best metric for batch inference?

Aggregate throughput, cost per completed job or token, and error rate are usually more important than single-request TTFT.

Can two LLM benchmark results be compared directly?

Only when the model, precision, hardware, request shape, load, serving configuration, and metric definitions are sufficiently comparable. Otherwise the numbers may describe different workloads.

How do I benchmark vLLM?

Current vLLM provides vllm bench serve for online-serving benchmarks, including TTFT, TPOT, ITL, end-to-end latency, throughput, percentiles, and goodput metrics.

Should model quality be part of an inference benchmark?

Performance and quality should both be evaluated when changing the model or its precision. A faster quantized model is only a useful improvement if it continues to meet the application's quality requirements.

How do I calculate LLM inference cost?

Once you know sustainable throughput under your latency SLO, divide the infrastructure cost by the amount of useful work completed. Cost per million output tokens is one common measure, while cost per successful request may be more useful for other applications.

What information should every LLM inference benchmark publish?

At minimum: model and revision, precision, hardware, GPU count, serving engine and version, input/output lengths, concurrency or request rate, benchmark duration, latency metrics, throughput, error rate, and any important limitations.

‍