PagedAttention and continuous batching: how vLLM serves more users

Running an LLM for one person is relatively simple.

Serving hundreds of requests at different prompt lengths, output lengths, and arrival times is a different systems problem.

One user may send a 500-token question and receive a 100-token answer. Another may send 20,000 tokens of documentation and ask for a long analysis. A third request may arrive halfway through both.

If the inference server treats those requests as one fixed batch and reserves memory for their worst possible size, two things happen quickly:

1. GPU memory gets wasted.
2. GPU compute sits idle while the server waits for uneven requests to finish.

PagedAttention and continuous batching address those problems from different directions.

PagedAttention manages the memory used by the KV cache.

Continuous batching manages the requests using the model.

vLLM combines both techniques as part of a serving system designed to keep GPU memory and compute more useful under changing workloads. Current vLLM still lists PagedAttention, continuous batching, chunked prefill, and prefix caching among its core serving capabilities.

Understanding the difference between them makes it much easier to reason about LLM throughput, concurrency, context length, and GPU utilization.

The short version

Here is the problem each technique solves:

‍

‍

These techniques interact.

They are not interchangeable.

A server can schedule requests brilliantly and still run out of KV-cache memory.

It can manage KV cache efficiently and still waste GPU compute with poor request scheduling.

Good LLM serving has to solve both.

Start with the problem PagedAttention was designed to solve

An autoregressive LLM stores key and value tensors for tokens it has already processed.

That is the KV cache.

Our KV cache guide goes through the memory calculation in detail, but the key point here is simple:

different requests need different amounts of KV-cache memory, and that amount changes while the request is running.

Suppose four requests enter the server:

Request A → 2K tokens
Request B → 14K tokens
Request C → 5K tokens
Request D → 31K tokens

Their caches are different sizes.

Then generation begins.

Request A may generate another:

300 tokens

while Request B generates:

6,000

and Request D continues for much longer.

The server therefore cannot know exactly how much cache every request will eventually need.

That unpredictability is awkward for memory allocation.

Why preallocating maximum KV-cache space wastes memory

One safe approach would be to reserve enough contiguous GPU memory for every request's maximum possible sequence length.

Imagine four requests all configured with a maximum length of:

32K tokens

The server could reserve four 32K cache regions:

Request A → [==============================]
Request B → [==============================]
Request C → [==============================]
Request D → [==============================]

But perhaps their actual active lengths are:

Request A → [==]
Request B → [==============]
Request C → [=====]
Request D → [============================]

Most of the reserved space for A and C is doing nothing.

That unused memory cannot easily be given to another request if the allocation model assumes each sequence owns a large contiguous region.

The original PagedAttention work identified fragmentation and redundant duplication of KV-cache memory as major constraints on batch size and therefore serving throughput.

PagedAttention borrows an idea from operating systems

Operating systems solved a related memory problem decades ago.

A process can behave as if it owns a large contiguous region of virtual memory even though the physical memory backing it is divided into smaller pages located elsewhere.

PagedAttention applies a similar idea to KV-cache storage.

Instead of requiring one large contiguous physical cache region per sequence, the cache is divided into blocks.

Conceptually:

Logical KV cache for Request A

[ block 0 ][ block 1 ][ block 2 ][ block 3 ]

            ↓ mapping ↓

GPU KV-cache memory

[ B2 ][ A0 ][ D4 ][ A1 ][ C0 ][ A2 ][ B3 ][ A3 ]

Request A's logical cache still behaves like one sequence.

Its physical blocks do not need to sit next to one another in GPU memory.

The original vLLM paper describes PagedAttention as an attention algorithm inspired by virtual memory and paging. The system allocates KV-cache blocks on demand and maintains mappings between logical sequence blocks and physical GPU-memory blocks.

Why blocks help

Suppose a sequence currently needs ten blocks.

The server allocates ten.

If the sequence grows enough to require an eleventh:

allocate one more block

rather than:

reserve enough memory for the entire theoretical sequence
from the beginning

When the request finishes, its blocks can be returned to the free pool and reused by other sequences.

That means memory follows actual demand much more closely.

It also reduces the problem of finding one enormous contiguous region of free GPU memory as requests enter and leave at different times.

PagedAttention does not make the KV cache smaller per token

This distinction matters.

If a model requires:

128 KiB of KV cache per token

PagedAttention does not suddenly make each token require:

32 KiB

The model architecture and cache precision still determine how much key/value state each token needs.

PagedAttention improves how that memory is allocated and reused.

It attacks:

fragmentation
over-reservation
duplication
allocation flexibility

rather than changing the fundamental per-token state the attention mechanism requires.

If you need to reduce the raw memory per cached token, techniques such as lower-precision KV cache address that different problem.

That distinction is why our KV cache article and this one belong together.

What did the original PagedAttention research find?

The original vLLM paper reported that its memory-management design produced near-zero waste in KV-cache memory and allowed flexible sharing of cache data within and across requests.

In the authors' evaluation at the time, vLLM achieved roughly 2–4× the throughput of the compared serving systems at a similar latency level, with larger gains appearing for longer sequences, larger models, and more complex decoding patterns.

Those numbers are historical research results from the 2023 system evaluation.

They should not be read as:

Installing today's vLLM makes every model four times faster.

Modern vLLM, competing runtimes, GPUs, attention kernels, quantization formats, and workloads have all changed.

The durable finding is the systems one:

using scarce GPU KV-cache memory more efficiently lets a server keep more useful sequences active.

Then there is the scheduling problem

Memory efficiency alone does not keep the GPU busy.

Imagine three requests batched together:

Request A → needs 20 output tokens
Request B → needs 200 output tokens
Request C → needs 2,000 output tokens

With a traditional run-to-completion batch, the group is awkward.

Request A finishes quickly.

Request B finishes later.

Request C keeps running.

If the server cannot change batch membership until the entire batch is done, capacity becomes stranded:

Step 1

A █
B █
C █

...

A finished
B █
C █

...

A finished
B finished
C █

The empty slots represent missed opportunities.

Another request may already be waiting.

Why fixed batching works badly for autoregressive generation

LLMs do not normally know in advance exactly how many tokens they will generate.

A user might ask for:

one sentence

while another asks for:

a detailed analysis

Even identical max_tokens settings do not mean requests will finish together.

One may emit an end-of-sequence token after 80 tokens.

Another may use the full allowance.

The result is a batch of requests with different lifetimes.

Traditional static batching works best when all items perform roughly the same amount of work.

Autoregressive generation does not offer that guarantee.

Continuous batching changes the batch while generation is running

Continuous batching operates at a finer scheduling granularity.

Instead of saying:

Build one batch and keep it unchanged until every request completes.

the server can effectively say:

Run this generation iteration, remove anything that finished, admit eligible waiting work, then run the next iteration.

Conceptually:

Iteration 1
[A B C]

Iteration 2
[A B C]

A finishes

Iteration 3
[D B C]

Iteration 4
[D B C]

B finishes

Iteration 5
[D E C]

The batch remains populated as requests come and go.

The earlier Orca serving system formalized this as iteration-level scheduling: the scheduler runs one model iteration at a time so completed requests can leave and new requests can join instead of waiting for a whole fixed batch.

vLLM currently lists continuous batching of incoming requests as one of its core serving capabilities.

Continuous batching is not the same as “send several prompts at once”

This distinction is worth keeping clear.

A static batch might contain:

100 requests

but remain static until all 100 finish.

That is batching.

It is not what makes continuous batching interesting.

Continuous batching allows the composition of active work to change between iterations.

So:

batching

answers:

Can several requests run together?

while:

continuous batching

answers:

Can the serving system continually reshape that active batch as requests arrive and finish?

The second property matters enormously for online traffic.

PagedAttention and continuous batching reinforce each other

Now the relationship becomes clearer.

Continuous batching wants to admit new requests whenever useful GPU capacity becomes available.

PagedAttention makes it easier to allocate KV-cache memory to those requests dynamically.

Together:

Request finishes
      ↓
KV blocks become free
      ↓
scheduler can admit waiting work
      ↓
new request receives cache blocks
      ↓
GPU continues processing a useful batch

If memory were badly fragmented or mostly trapped inside oversized reservations, the scheduler might know that another request could use compute while still lacking usable cache space to admit it.

If memory were managed perfectly but the scheduler could not replace completed requests, some of that available capacity would still sit idle.

This is why the two techniques belong in the same serving story.

Prefill and decode are different workloads

There is another complication.

An LLM request has two broad phases.

Prefill

The server processes the prompt:

"Here are 12,000 tokens of context..."

This stage handles many tokens at once.

Decode

The model begins autoregressive generation:

token 1
token 2
token 3
...

Each active sequence typically advances incrementally.

vLLM's current optimization guidance describes prefill as more compute-bound and decode as more memory-bound.

That difference creates another scheduling challenge.

A huge prefill can interfere with active generation

Imagine several users already receiving streamed answers.

Their decode steps are relatively small and frequent.

Then a new request arrives with:

80K-token prompt

Processing the entire prefill as one enormous unit could occupy substantial GPU compute.

Existing users may then experience worse inter-token latency while their decode work waits.

The new user wants a low time to first token.

Existing users want smooth generation.

The scheduler has to balance both.

Chunked prefill splits large prompts into schedulable pieces

Modern vLLM addresses this with chunked prefill.

Instead of requiring the entire long prompt to be processed in one scheduling step, a prefill can be divided into smaller token chunks.

That lets the scheduler mix:

decode work
+
part of a long prefill

within its token budget.

Current vLLM V1 enables chunked prefill by default whenever possible. Decode requests are prioritized, then available token budget is used for waiting prefills; a prefill that does not fit can be split into chunks.

Conceptually:

Without chunking

[============ huge prefill ============]
                                   [decode]

With chunking

[prefill][decode][prefill][decode][prefill][decode]

The real scheduler is more sophisticated than that diagram.

The principle is what matters.

Chunked prefill is not continuous batching either

We now have three related ideas:

PagedAttention
Manage KV-cache allocation.

Continuous batching
Change active request membership between inference iterations.

Chunked prefill
Split large prompt-processing work so it can be scheduled alongside other requests.

They cooperate.

They solve different problems.

This is the kind of distinction that often disappears when someone says:

vLLM is faster because of PagedAttention.

PagedAttention was foundational to vLLM's design.

Modern serving performance comes from an entire system of memory management, scheduling, kernels, quantization, caching, compilation, parallelism, and other optimizations. Current vLLM lists all of these separately rather than treating PagedAttention as its only acceleration technique.

Why chunked prefill can improve both throughput and latency

vLLM's current scheduler prioritizes pending decode work and uses remaining batch-token capacity for prefill.

Its documentation identifies two main benefits:

• better inter-token latency because decode work is prioritized
• better GPU utilization by combining compute-heavy prefill work with memory-heavy decode work where possible

There is still a tradeoff.

Giving more of an iteration's token budget to prefill can improve time to first token for new requests.

Giving less can protect decode smoothness for existing requests.

Current vLLM exposes max_num_batched_tokens as one of the controls for that balance. Smaller token budgets can favor inter-token latency, while larger budgets can favor prefill speed and throughput.

That setting should be tuned against your workload rather than copied from somebody else's benchmark.

What does max_num_seqs control?

vLLM also exposes:

max_num_seqs

which limits how many sequences can be processed in one scheduler iteration.

Conceptually:

max_num_seqs = 64

means the scheduler cannot run an arbitrarily large number of active sequences in one iteration even if thousands are queued.

The right number depends on:

• model
• GPU memory
• KV-cache capacity
• prompt lengths
• output lengths
• latency target
• batch-token budget

More concurrent sequences can improve throughput.

They also require more cache and may increase contention.

This is another reason “maximum GPU utilization” is not a useful serving objective by itself.

What does max_num_batched_tokens control?

This setting limits the number of tokens the scheduler can process in one iteration.

That is different from the number of sequences.

You could have:

many sequences
×
small token contribution

or:

fewer sequences
+
a large prefill

and reach the same token budget.

This gives the scheduler a way to bound the work assigned to one iteration rather than reasoning only in terms of request count.

For real deployments, token workload is often more informative than request count.

A hundred 200-token prompts and a hundred 100K-token prompts are both:

100 requests

but they are not remotely equivalent workloads.

Why online serving needs scheduling rather than one perfect batch size

There is no single batch size that represents live traffic.

At 09:00 you may have:

3 active requests

At 09:01:

80

Then several finish.

New long-context requests arrive.

Others ask for one-token classifications.

A production scheduler is continually solving:

What can I run now?

What fits in cache?

What is waiting?

Which requests are decoding?

Which requests need prefill?

How much work can fit in this iteration?

That is why LLM serving is fundamentally a scheduling problem as well as a model-execution problem.

Why higher throughput can hurt latency

Imagine your only objective is:

maximum total tokens per second

The server could try to keep very large batches in flight.

That may use the GPU efficiently.

Individual users may wait longer before receiving their first token.

Or tokens may arrive less smoothly once generation begins.

Conversely, optimizing aggressively for:

minimum latency for one request

can leave the GPU underutilized and reduce total throughput.

The interesting metrics are therefore not interchangeable.

TTFT, TPOT, ITL, and throughput

vLLM currently exposes and benchmarks several useful latency measures.

Time to first token (TTFT)

How long the user waits before the first generated token arrives.

TTFT includes the effect of prompt processing and scheduling delay.

Long prefills can make this large.

Time per output token (TPOT)

Average generation time per output token after the first.

This is useful for understanding decode performance.

Inter-token latency (ITL)

The delay between successive streamed output tokens.

For a conversational application, this affects how smooth generation feels.

Throughput

How much total work the server completes over time.

Depending on the metric, this might be:

requests / second

or:

tokens / second

You need more than one of these numbers to understand whether a serving configuration is actually good.

A throughput number without latency can be misleading

Suppose configuration A achieves:

2,000 tokens/s

and configuration B:

2,600 tokens/s

B appears better.

But perhaps its P99 TTFT jumps from:

500 ms

to:

8 seconds

If you are running an offline batch job, that trade might be acceptable.

If you are serving an interactive assistant, it may be terrible.

This is why vLLM's current benchmark tooling reports percentile metrics for TTFT, TPOT, ITL, and end-to-end latency rather than only total throughput.

What is goodput?

Throughput counts completed work.

Goodput asks how much completed work also met the latency requirements you actually care about.

Imagine:

Server A:
100 requests/s
but 40% violate latency target

Server B:
85 requests/s
and 98% meet latency target

Raw throughput favors A.

A user-facing service may prefer B.

Current vllm bench serve supports goodput criteria based on latency objectives such as TTFT and TPOT.

That is a better way to think about production performance than chasing one maximum tokens-per-second number.

PagedAttention and FlashAttention are not the same thing

The names are similar enough to create regular confusion.

PagedAttention is about managing the KV cache and accessing cache blocks that do not need to occupy one contiguous physical region.

FlashAttention is an optimized attention-computation approach designed to reduce expensive memory movement during the attention calculation itself.

They solve different problems.

Modern vLLM can use optimized attention backends such as FlashAttention while also using its own KV-cache memory-management architecture. The current vLLM feature set lists PagedAttention and optimized attention kernels as separate capabilities.

So this is perfectly reasonable:

PagedAttention
+
FlashAttention

They are complementary.

PagedAttention and prefix caching are not the same thing either

PagedAttention lets the serving system manage KV-cache blocks efficiently.

Prefix caching goes further by recognizing when requests share eligible previously computed prefixes and reusing those cache blocks.

Imagine every request starts with the same:

20K-token system prompt

Without prefix reuse, the server may repeatedly calculate that prefix.

With prefix caching, eligible repeated prefixes can reuse already computed KV state.

Paged block management makes this kind of sharing easier, and the original PagedAttention paper explicitly discusses flexible KV-cache sharing within and across requests.

But the concepts remain different:

PagedAttention
→ how cache blocks are represented and managed

Prefix caching
→ when previously computed blocks can be reused

PagedAttention does not eliminate out-of-memory errors

Efficient allocation gives you more usable memory.

It does not give you unlimited memory.

If the GPU has enough cache capacity for:

200K active tokens

and your workload tries to maintain:

400K active tokens

something has to give.

Requests may queue, be preempted, or fail depending on the serving configuration.

Current vLLM documentation explicitly discusses request preemption when there is insufficient KV-cache space and suggests options such as reducing active sequence/token limits or increasing model parallelism to free additional cache memory.

Paging removes waste.

It does not repeal capacity limits.

Continuous batching does not guarantee lower latency for every request

Continuous batching primarily improves the server's ability to keep useful work in flight.

Under load, that can substantially improve throughput and utilization.

But allowing more requests to share the GPU also means more competition for compute and cache.

A request running alone may have lower latency than the same request sharing a heavily loaded server.

The goal is not:

make every individual request faster

It is:

use the GPU efficiently
while meeting the service's latency targets

That distinction matters when evaluating any batching claim.

Why the original Orca idea mattered

Before modern LLM serving frameworks, batch processing often assumed that all work in a batch would complete together.

Orca recognized that autoregressive transformer inference behaves differently.

Its iteration-level scheduler allowed the system to reconsider which requests were active after each generation iteration instead of locking one batch for the duration of every sequence.

On a GPT-3 175B evaluation, the original Orca paper reported large throughput improvements over the serving baseline it compared against at similar latency.

Again, the exact historical multiplier is less useful today than the architectural insight:

generation is iterative, so scheduling should be iterative too.

Continuous batching builds on that way of thinking.

Why continuous batching helps GPU utilization

A GPU likes enough parallel work to keep its compute resources busy.

If active batch membership shrinks like this:

32 requests
↓
19
↓
8
↓
2
↓
1

and the server cannot add new work until the final request finishes, utilization can collapse toward the tail of the batch.

Continuous batching can instead behave more like:

32
↓
request finishes
↓
replace it
↓
32
↓
another finishes
↓
replace it
↓
32

subject to cache, scheduling, and token-budget constraints.

The batch is not literally guaranteed to remain full at every instant.

The scheduler simply has the opportunity to keep using capacity instead of waiting for unrelated requests to finish.

Memory and scheduling are linked through admission control

Suppose the scheduler has ten requests waiting.

The GPU has compute capacity.

Can it admit all ten?

Only if the KV-cache manager can provide enough memory for them.

So admission is constrained by both:

compute budget
+
memory budget

This is where PagedAttention and continuous batching become inseparable in practice.

The scheduler wants to keep the machine busy.

The memory manager determines how much active state can remain resident.

What happens when KV cache is exhausted?

Current vLLM can preempt requests when KV-cache space becomes insufficient.

Its optimization guidance notes that repeated preemption and recomputation can hurt end-to-end latency and recommends changes such as reducing max_num_seqs or max_num_batched_tokens, increasing available cache memory, or distributing model weights across more GPUs to leave additional VRAM for cache.

That tells us something important about tuning:

higher concurrency stops being useful when it causes memory pressure severe enough to repeatedly disrupt active requests.

Maximum admission is not the same as maximum useful throughput.

This connects directly to tensor parallelism

Suppose model weights occupy most of one GPU.

The server has very little room left for KV cache.

Moving to two-way tensor parallelism can shard those weights across two GPUs.

That may leave much more cache space on each card.

Now PagedAttention has a larger memory pool to manage, and continuous batching can potentially keep more requests active.

This is why model parallelism can improve serving capacity even when the primary goal is not simply “make the model fit.”

The memory and scheduling layers interact.

And it connects directly to quantization

The same logic applies to NVFP4 quantization.

Suppose:

FP8 model weights = 28GB

on a:

32GB GPU

There is very little room left.

If an NVFP4 checkpoint reduces the model to:

18GB

the freed memory can become KV-cache capacity.

That could mean:

longer context
more sequences
more concurrency

even if the model was already technically able to load before quantization.

Quantization, PagedAttention, and batching solve different problems.

Their benefits compound.

How should you tune a vLLM endpoint?

Start with the workload rather than the flags.

Measure:

• prompt-length distribution
• output-length distribution
• request arrival rate
• target TTFT
• target ITL or TPOT
• desired concurrency
• GPU-memory use
• queue time
• preemption rate

Then adjust the scheduler.

Current vLLM exposes controls including:

max_num_seqs
max_num_batched_tokens
scheduling policy
chunked prefill

among others.

Do not start by copying a configuration from a benchmark using a different model, GPU, and request distribution.

Benchmark the traffic you actually expect

vLLM includes:

vllm bench serve

for testing online serving performance. Current versions can report TTFT, TPOT, ITL, throughput, and selected latency percentiles.

A simple synthetic run can look like:

vllm bench serve \
 --model <your-model> \
 --host <server-host> \
 --port 8000 \
 --dataset-name random \
 --random-input-len 1024 \
 --random-output-len 256 \
 --num-prompts 500

That is useful for controlled comparisons.

It is still synthetic.

If your actual application sends:

20K prompts
+
4K outputs

a benchmark using:

1K prompts
+
256 outputs

may lead you toward the wrong scheduler configuration.

Use representative traffic whenever possible.

Change one serving variable at a time

Suppose you want to test:

max_num_batched_tokens

Keep constant:

• model
• checkpoint
• precision
• GPU count
• request-rate pattern
• prompt lengths
• output lengths
• context limit

Then change the token budget.

Compare:

TTFT
ITL
throughput
queue time
preemptions

If you simultaneously change quantization, tensor parallelism, context length, and scheduler settings, you will know that performance changed.

You will not know why.

The best batch is workload-dependent

For an offline job, you may prioritize:

maximum throughput

and tolerate high individual latency.

For an interactive assistant, you may prioritize:

low TTFT
smooth ITL

while accepting somewhat lower total token throughput.

For an internal coding service, long prompts may dominate.

For a classification endpoint, outputs may be one or two tokens.

Those systems should not use identical serving configurations merely because they run the same base model.

The workload is part of the model server.

What does this mean for one RTX 5090?

An RTX 5090 has finite VRAM.

Suppose a quantized model fits comfortably enough to leave substantial cache memory.

PagedAttention helps the server use that cache efficiently across requests of different lengths.

Continuous batching helps it keep processing useful requests as some finish and others arrive.

Chunked prefill helps long prompts coexist with active decoding.

That combination is one reason a single GPU can support a much more useful inference service than a naïve:

for prompt in requests:
   model.generate(prompt)

loop would suggest.

For the hardware side, our RTX 5090 VRAM guide explains how model weights and KV cache compete for the same memory.

For actual serving setup, use the Hivenet vLLM guide.

When should you use vLLM?

vLLM becomes particularly useful when the workload has moved beyond one person experimenting with one model.

Its current serving stack combines techniques including:

• PagedAttention
• continuous batching
• chunked prefill
• prefix caching
• quantization
• optimized attention kernels
• speculative decoding
• distributed parallelism
• OpenAI-compatible serving

That makes it a strong fit for self-managed production inference where throughput, latency, concurrency, and model configuration matter.

If you are still choosing an inference engine, our vLLM vs TGI vs TensorRT-LLM vs Ollama comparison covers the broader decision.

If you do not want to manage an inference server at all, Hivenet Inference API solves a different problem: consuming an inference endpoint rather than operating the serving infrastructure.

The useful way to think about PagedAttention and continuous batching

PagedAttention asks:

How do we keep the KV cache from wasting scarce GPU memory as sequences grow and shrink?

Continuous batching asks:

How do we keep useful requests flowing through the GPU as requests arrive and finish at different times?

Chunked prefill asks:

How do we stop one enormous prompt from monopolizing the schedule while existing requests are decoding?

Together, they turn LLM inference from:

load model
→ send batch
→ wait
→ send next batch

into a continuously managed system.

Requests arrive.

Memory blocks are allocated.

Prompts are processed.

Sequences decode.

Some finish.

Their memory is released.

New work enters.

The scheduler keeps going.

That is the part of vLLM that matters more than the phrase “fast inference.”

It is a system designed around the fact that real LLM requests are uneven.

PagedAttention and continuous batching FAQ

What is PagedAttention?

PagedAttention is a KV-cache memory-management approach introduced with vLLM. It divides cache storage into blocks and maps a sequence's logical cache blocks to physical GPU-memory blocks rather than requiring one large contiguous allocation.

Why is PagedAttention useful?

LLM sequences grow to different lengths and finish at different times. Block-based allocation reduces fragmentation and unnecessary cache reservation, allowing GPU memory to accommodate more useful active sequences.

Does PagedAttention reduce KV-cache memory per token?

No. It primarily improves allocation and reuse. The model architecture, sequence length, KV-head count, head dimension, and cache precision still determine the underlying amount of KV state required.

Is PagedAttention the same as FlashAttention?

No. PagedAttention addresses KV-cache memory management. FlashAttention optimizes the attention computation and its memory-access pattern. Modern serving stacks can use both.

What is continuous batching?

Continuous batching lets a serving system add or remove requests between model-generation iterations rather than keeping one fixed batch until every sequence finishes. The underlying iteration-level scheduling idea was established by systems such as Orca and is used by modern LLM servers such as vLLM. citeturn144773search2turn144773search0

Why is continuous batching better than static batching for LLMs?

LLM outputs have unpredictable lengths. In a fixed batch, short requests can finish while long ones keep running, leaving unused batch capacity. Continuous batching lets new work replace finished sequences instead of waiting for the longest request.

Is continuous batching the same as dynamic batching?

The terms are sometimes used loosely. The important property in LLM serving is iteration-level admission and removal: the active batch can change between autoregressive generation steps rather than remaining fixed for the entire request.

What is chunked prefill?

Chunked prefill splits a large prompt-processing operation into smaller pieces so the scheduler can mix prefill work with ongoing decode work. Current vLLM V1 enables chunked prefill by default whenever possible.

Is chunked prefill the same as continuous batching?

No. Continuous batching changes which requests are active between iterations. Chunked prefill changes how a large prompt is divided into schedulable work. They complement each other.

What is prefill in LLM inference?

Prefill is the phase where the model processes the input prompt and creates the initial attention/KV state before autoregressive output generation begins.

What is decode?

Decode is the autoregressive generation phase where the model repeatedly generates additional output tokens using the prompt and previously generated state.

Why does vLLM prioritize decode during chunked prefill?

Current vLLM scheduling prioritizes pending decode work and uses remaining batch-token budget for prefill. Its documentation says this can improve inter-token latency while combining compute-heavy prefill and memory-heavy decode workloads more effectively.

What does max_num_seqs do in vLLM?

It limits the maximum number of sequences that can be processed in one scheduler iteration.

What does max_num_batched_tokens do?

It limits the number of tokens that can be scheduled in one iteration and is an important control for balancing prefill performance, decode latency, and throughput.

What is TTFT?

TTFT means time to first token: how long a request waits before its first generated token is produced. Current vLLM exposes TTFT as a production and benchmark metric.

What is ITL?

ITL means inter-token latency: the delay between successive generated tokens. It is particularly relevant to streamed interactive responses.

Does higher throughput mean lower latency?

No. A configuration can increase total throughput by running more work concurrently while increasing waiting time or per-request latency. Both throughput and latency percentiles should be measured.

What is prefix caching?

Prefix caching reuses eligible previously computed KV-cache blocks when requests share the same prefix, reducing repeated prefill computation. It is separate from PagedAttention itself.

Can PagedAttention prevent every OOM error?

No. It reduces waste in KV-cache allocation, but the GPU still has finite memory. Too many active tokens or sequences can exhaust available cache capacity.

Does vLLM still use PagedAttention?

Yes. Current vLLM documentation continues to list efficient KV-memory management with PagedAttention as a core feature, alongside continuous batching, chunked prefill, prefix caching, quantization, and other serving optimizations.

How do I benchmark PagedAttention or batching settings?

Use a consistent serving workload and compare metrics such as TTFT, TPOT, ITL, request throughput, token throughput, queue time, and preemptions. Current vLLM provides vllm bench serve for online-serving benchmarks.

‍