Continuous batching explained for LLM inference, explained

Real traffic is messy. New requests arrive while others are mid-generation. In autoregressive LLM serving, keeping a batch fixed until its longest response finishes leaves completed requests unable to make room for new work. Continuous batching lets the server update the active batch between generation steps.

The scheduler can admit waiting requests when capacity becomes available, rather than waiting for every request in a fixed batch to finish. This is iteration-level scheduling. It does not mean reloading model weights for each token or request: in a typical GPU-resident deployment, weights remain loaded while the server processes many requests. Throughput gains depend on the model, hardware, traffic and serving configuration.

Prefill vs decode

Prefill processes the prompt tokens and builds their key/value cache. Decode uses that cached state to generate subsequent tokens autoregressively. The prompt is the input, not an extra sequence alongside it.

Prefill can process many prompt tokens in parallel. A serving engine may split long prompts into chunks and interleave them with decode work. Prefill is often compute-intensive; small-batch decode is often limited by memory bandwidth. The bottleneck depends on the workload and hardware.

Continuous batching is a serving-engine scheduling technique and does not require retraining the model. PagedAttention addresses a separate problem: managing the KV cache in blocks that need not be contiguous. These techniques can work together, but neither is a synonym for the other.

Review Compute for your workload

Check current hardware, locations, pricing and deployment requirements before choosing an instance.

Try Compute now

How continuous batching works

• Admit work between iterations. When token, request and memory budgets allow, the scheduler adds waiting requests without waiting for the whole batch to finish.
• Manage KV-cache blocks. Each request needs the correct state for its own token sequence. Engines can reuse memory after requests finish; matching-prefix caching is a separate optimization. vLLM’s prefix-caching documentation also describes isolation controls for shared deployments.
• Grow and shrink on the fly. As requests finish or get canceled, the active batch size changes without forcing a restart. This dynamic approach optimizes memory usage and memory bandwidth by allocating resources as needed.
• Stream when enabled. A streaming API can deliver generated tokens while the scheduler handles other requests. Streaming controls delivery to clients; continuous batching controls model execution.

Batching can spread work across more requests per model iteration, but higher total throughput does not guarantee lower latency for every user. Compare both queueing delay and token-delivery latency under the same workload.

How it affects TTFT and TPS

• Less fixed-batch waiting. Waiting requests can join when a slot becomes available. Time to first token also depends on queueing and prefill, so admission policy matters.
• Higher aggregate utilization. Replacing finished requests can improve server-wide output throughput when response lengths differ. Measure per-request latency separately.
• Overload still needs control. Continuous batching cannot remove sustained excess demand. Set queue limits and timeouts and test behavior beyond expected peak load.

Limits and trade‑offs

• Memory pressure. Model weights, runtime workspaces and the KV cache compete for GPU memory. Long contexts and many active requests increase cache demand; insufficient headroom can cause preemption, recomputation or out-of-memory errors.
• Fairness vs speed. Aggressive packing can starve long prompts. Proprietary models may have different maximum context length limits, which affects batching strategies and how user input of varying lengths is handled. Add simple fairness rules and output caps.
• Jitter. Streaming interleaves many users. Variable user input lengths can contribute to token timing variation. Token timing varies a bit, which is fine for chat but less so for hard real‑time tasks. Continuous batching requires sophisticated scheduling algorithms and robust backend systems to manage variable workload processing.

Tuning checklist

This section describes key steps to create an efficient batching setup for your models.

Start with defaults, then change one thing at a time:

1. Cap max_tokens. Set a useful output limit and test how the engine schedules it. A high ceiling permits long generations; it does not necessarily allocate the full maximum cache in advance.
2. Set a sane context length. Use RAG to keep prompts short when you can.
3. Pick batch limits. Allow enough parallel decode slots to keep the GPU busy, but avoid thrashing.
4. Right-size hardware. Check queueing, cache occupancy, preemptions and memory use before changing the GPU or model. A low prefix-cache hit rate can simply mean prompts share little text.
5. Stream when useful. Streaming can show progress sooner, but does not by itself reduce the model work or queue length.
6. Use short system prompts. Fewer tokens in, fewer tokens to cache.

Tests to run

For thorough validation, it is recommended to run these tests over several iterations, using different iterations and multiple iterations to cover a range of scenarios. For example, you can vary prompt lengths or concurrency levels in each run to observe system behavior.

• Load ramp. Increase concurrency step by step and watch TTFT and TPS.
• Mixed prompts. Combine short and long prompts to see how the scheduler behaves.
• Cancel storms. Fire many cancels to check cleanup and cache reuse.
• Token caps. Verify that caps work and errors are readable.
• Hot reload. Restart the server or reload a model and confirm recovery.

Instrumentation that pays off

Track these at minimum:

• TTFT and inter-token latency p50/p95, plus aggregate output TPS under load
• Active batch size and queue length
• GPU memory use, KV-cache occupancy and preemptions; prefix-cache hit rate if enabled
• Error codes: OOM, timeouts, 5xx
• Per-request prompt and output token counts; avoid raw prompt and response logs unless justified

Before deciding what your inference server records, use our privacy checklist for LLM inference in Europe to review logging, retention, access, and the full data path.

Wrap‑up

Continuous batching is not magic. It is a practical way to keep GPUs busy and users happy when traffic is uneven. Start with safe caps, measure TTFT and TPS, and adjust batch limits where the numbers say it matters.

Plan a self-managed deployment
Use the existing vLLM endpoint on Compute walkthrough as a reference, not confirmation that its image or configuration is currently available. Check current Compute documentation and available hardware, and configure endpoint authentication and TLS for your deployment.

FAQ

What is prefill vs decode?

Prefill processes prompt tokens into cached attention state, potentially in chunks. Decode then generates tokens using that state.

How big should a batch be?

Big enough to keep the GPU busy during decode without causing cache thrash. Test with your real prompts and cap max tokens.

Why does TTFT get worse at high load?

Queueing and prefill work often dominate at high load. Inspect prompt lengths, concurrency, preemptions and scheduler settings; cache hit rate alone cannot diagnose the cause.

Does continuous batching work with streaming?

Yes. Streaming is the default for many servers. Users see tokens while the scheduler keeps admitting other requests.

Do I need multi‑GPU for this to help?

No. Continuous batching can be useful on one GPU when multiple requests are in flight. Multiple GPUs may be needed for model capacity or throughput, but the benefit must be measured.

How many words is 1,000 tokens?

For English prose, 1,000 tokens is sometimes approximated as 750 words. Do not use that shortcut to size a request: count tokens with the model’s tokenizer, since language and text type change the ratio.

What does a token mean in AI?

A token is a unit of text used in natural language processing, often a word or subword piece that the model processes during inference or training.

What is TTFT?

TTFT means time to first token. From the client’s perspective, it runs from request submission to receipt of the first output token. Server-side timings use a different boundary.

How to measure TTFT?

For a streaming request, time from submission to the first non-empty token-bearing response. Record the measurement boundary and keep it consistent across tests.

What is the TPOT metric?

TPOT means time per output token, a latency measure, usually expressed in milliseconds per token. A common per-request calculation excludes the first token: (total latency − TTFT) / (output tokens − 1), for responses longer than one token. Check the benchmark tool’s definition.

What is TPS in LLM?

TPS means tokens per second. State whether you mean one response or total output across concurrent requests; the two can move in opposite directions as load increases.

What is a KV cache?

KV cache refers to the key-value cache that stores intermediate key and value tensors computed during the attention mechanism to speed up subsequent token generation.

What is GPU KV cache?

GPU KV cache is the storage of key-value pairs in GPU memory used during model inference to optimize attention computations and reduce redundant calculations.

What is the KV cache in LLM?

In large language models, the KV cache holds cached key and value vectors from previous tokens to efficiently compute attention for new tokens without recalculating past states.

What is key-value store cache?

A general key-value store retrieves values by keys. An LLM KV cache instead retains attention key and value tensors for token positions. The shared abbreviation does not make them the same system.

How does dynamic batching work?

Request-level dynamic batching groups compatible requests before execution, with an optional wait to form a larger batch. Triton’s dynamic batcher is intended for stateless models. Continuous LLM batching instead revisits the active set between generation iterations.

What is the difference between static batching and dynamic batching?

A static batch has fixed membership for an execution. A dynamic batch is assembled from queued requests using configured limits and, optionally, a waiting window. Continuous batching adds a different capability: changing active requests between generation iterations. Static batching does not inherently require waiting for a completely full batch.

‍