Multi-GPU LLM serving: Architecture and scaling guide

Multi-GPU LLM serving is useful for two different problems: one model replica needs more memory than a single GPU provides, or one replica cannot meet a measured throughput target. Those problems call for different designs. Shard one model with tensor or pipeline parallelism when it cannot fit. Run independent replicas when the model already fits and traffic is the constraint.

Adding GPUs is not a free speedup. Sharded inference introduces communication and synchronization on the token path. More aggregate VRAM does not become one flat memory pool, and a configuration that loads successfully can still have worse time to first token or lower cost efficiency. Start with the smallest design that meets the workload, then scale from evidence.

Key takeaways

• Use one GPU while the model, runtime, and expected KV cache fit with safe headroom and the server meets its latency and throughput targets.
• Use tensor parallelism when one model must be split across tightly connected GPUs. Communication topology can decide whether it helps.
• Use pipeline parallelism when layers need to be divided into stages, especially across node boundaries, but account for pipeline bubbles and stage imbalance.
• Use independent replicas for throughput when a complete model fits on each replica. This usually offers simpler isolation and failure recovery than sharding every request.
• Tune batching and KV-cache limits before buying more capacity. Long prompts, loose output caps, and mixed request shapes can exhaust memory or inflate tail latency.
• Compare systems with identical model revisions, precision, prompts, output limits, concurrency, and latency objectives. A peak tokens-per-second number is not enough.

Decide whether you need multiple GPUs

Reason 1: the model does not fit

Start with a memory budget, not a parameter-count shortcut. The deployment must hold model weights, runtime allocations, temporary workspaces, and the KV cache for active requests. Quantization can reduce weight memory, while grouped-query attention, cache precision, context length, and concurrency affect KV-cache demand. Keep additional headroom for uneven allocation and traffic bursts.

If a supported quantized model preserves acceptable quality, it may avoid sharding altogether. The LLM quantization guide covers that evaluation. If the model still does not fit, split it with a serving engine that supports the architecture and chosen precision.

Reason 2: one replica cannot meet the service objective

A model can fit comfortably and still miss its production target. Measure time to first token (TTFT), time per output token (TPOT), end-to-end latency, throughput, queue time, error rate, and GPU-memory headroom under representative concurrency. Our LLM inference metrics guide explains how to read those values without collapsing them into one speed number.

If a single replica meets latency at low traffic but queues grow under load, add replicas before sharding the model. Each replica can schedule requests independently, and a load balancer can route around an unhealthy worker. Sharding is appropriate when each request genuinely needs several GPUs or when a tested sharded replica provides the better latency-throughput tradeoff.

Choose the parallelism strategy that matches the bottleneck

Strategy	What is split	Use it when	Main cost
Tensor parallelism	Operations and weight tensors inside each layer	A model needs several closely connected GPUs or one sharded replica benchmarks well	Frequent collective communication on the inference path
Pipeline parallelism	Contiguous groups of layers into stages	The model spans nodes or layer staging fits the topology better	Pipeline bubbles, stage imbalance, and activation transfer
Independent replicas	Requests across complete copies of the model	The model fits on each replica and traffic is the bottleneck	Duplicate weight memory and load-balancing work
Expert parallelism	Experts in a Mixture-of-Experts model	The model architecture and serving engine support distributed experts	Token routing, all-to-all traffic, and uneven expert load

Tensor parallelism

Tensor parallelism divides work within transformer layers. Each GPU stores and computes a shard, then exchanges partial results with the other ranks. That is why GPU count alone is a weak predictor. The communication path between ranks, collective implementation, tensor-parallel degree, model architecture, and batch shape all affect the result.

Current vLLM parallelism documentation supports tensor- and pipeline-parallel serving and configures tensor parallelism with the desired number of ranks. NVIDIA’s TensorRT-LLM guidance likewise treats communication cost as the central sharding decision. Read our tensor-parallelism explainer for the layer-level mechanics.

Pipeline parallelism

Pipeline parallelism places different groups of layers on different stages. Activations move from one stage to the next instead of every rank participating in the same layer computation. That can reduce some collective traffic across slower boundaries, but a stage can sit idle while it waits. Uneven layer cost creates further imbalance, and microbatching that improves utilization can change latency.

A common starting point for multi-node serving is tensor parallelism within a node and pipeline parallelism across nodes. Treat that as a hypothesis to benchmark, not a universal rule. The best split depends on the actual interconnect, model, runtime, and request shape.

Independent replicas and data parallel serving

For dense models that fit on one GPU or one sharded node, independent replicas are usually the clearest way to add throughput. Each replica loads a complete model and handles different requests. Current vLLM deployment guidance describes internal, hybrid, and external load-balancing patterns and notes that non-MoE models can use independent serving instances behind an external router.

Scale on queue time, latency, and sustained utilization rather than GPU utilization alone. A busy GPU can be healthy, while a lightly utilized deployment can still suffer from long prompts, CPU tokenization, network delay, or an uneven request router.

Expert parallelism for MoE models

Expert parallelism applies to Mixture-of-Experts models. Instead of slicing every expert across every rank, the runtime can place complete experts on different GPUs and route tokens to them. This can improve the memory and compute arrangement for a supported MoE model, but it adds all-to-all communication and can suffer when some experts receive far more tokens than others. Do not apply expert-parallel guidance to a dense transformer.

Topology can matter more than the GPU label

Tensor-parallel requests exchange data repeatedly, so ranks connected through a fast local fabric behave differently from ranks connected only through a weaker PCIe path or a network between hosts. NVIDIA’s NCCL documentation describes topology-aware collective communication across PCIe, NVLink, InfiniBand, and IP networking.

Before serving, inspect which GPUs can communicate directly, confirm that the container or virtual machine exposes the intended topology, and run a collective-communication benchmark across the exact ranks you plan to use. Then benchmark inference. An AllReduce test can reveal a bad path; it cannot prove that a particular model, context length, or serving engine will meet its latency target.

Multi-node deployments also expand the security boundary. vLLM warns that its distributed runtime traffic is unencrypted. Keep worker communication on a trusted private network, restrict access, and do not expose distributed-control or data-plane ports to untrusted clients.

Batching and KV cache determine useful capacity

During autoregressive generation, the KV cache stores attention state for active sequences so the server does not recompute the full history for every new token. Its memory use grows with active tokens and model-specific dimensions. Longer contexts and more concurrent requests can therefore consume the memory that appeared free after loading the weights. The KV-cache guide shows the calculation and explains why aggregate VRAM is not the whole capacity plan.

PagedAttention and continuous batching solve related but distinct serving problems. PagedAttention manages KV-cache blocks so allocation can follow sequences as they grow. Continuous batching lets the scheduler admit and retire requests between generation steps. The original PagedAttention paper connects more efficient cache allocation with the ability to batch useful work.

Current vLLM scheduler controls include limits for concurrent sequences and batched tokens, plus chunked-prefill behavior and KV-cache admission headroom. Raising a limit can increase throughput until cache pressure, preemption, or tail latency erases the gain. Tune with a concurrency sweep using the real prompt and output-length distribution.

A practical scaling sequence

1. Freeze the workload. Record the exact model revision, tokenizer, precision, maximum context, prompt distribution, output cap, concurrency range, and latency objective.
2. Establish the single-GPU baseline. Measure warm and cold startup, TTFT, TPOT, end-to-end latency, throughput, queue time, error rate, and memory headroom.
3. Reduce avoidable demand. Test supported quantization, realistic context caps, prompt caching where applicable, and shorter output limits before changing topology.
4. Choose sharding or replication. Shard because one request needs several GPUs; replicate because traffic needs more independent capacity.
5. Validate communication. Check the exposed topology and collective bandwidth for the exact GPU group. Repeat after driver, runtime, image, or host changes.
6. Sweep scheduler limits. Change one batching or cache control at a time and retain the latency curve, not only the fastest result.
7. Test failure behavior. Restart a worker, cancel streams, overload the queue, and verify health checks, request timeouts, draining, and retry safety.
8. Compare economics. Calculate cost per completed request or million output tokens while meeting the latency objective. Include idle replicas and operational work.

Benchmark the serving system, not the GPU name

Question	Evidence to keep
Does the model fit safely?	Weight precision, runtime allocation, KV-cache budget, peak memory, and OOM behavior
Is interaction responsive?	TTFT, TPOT or inter-token latency, end-to-end latency, and percentiles
Does capacity scale?	Throughput, goodput, queue time, and error rate across a concurrency sweep
Is communication healthy?	Topology map, collective bandwidth, communication time, and cross-node network details
Is the result economical?	Price basis, utilization, completed requests, useful tokens, and operating overhead

Keep warmup, test duration, sampling parameters, input and output lengths, model revision, runtime version, driver, GPU topology, and region constant when comparing configurations. The broader production inference guide covers deployment, safety, and operating controls, while the serving-engine comparison helps choose the runtime before tuning it.

How Hivenet fits multi-GPU serving

Compute with Hivenet currently provides GPU or CPU instances where the customer controls the operating system, serving engine, model, dependencies, and deployment. The current product page lists RTX 5090 and RTX 6000-series paths, per-second billing, and regional deployment options. Availability, GPU count, configuration, and price can change, so verify the current console or discuss a larger configuration before designing around it. Teams that want an operated OpenAI-compatible endpoint instead of infrastructure should evaluate Hivenet Inference rather than assuming Compute is a managed vLLM service.

Hivenet’s benchmark hub reports a bounded single-host test in which an eight-GPU RTX 5090 virtual machine matched the bare-metal NCCL AllReduce baseline within normal run-to-run variance. That supports the tested VM communication path. It does not guarantee identical model-serving performance, and it does not cover cross-host inference. Use the same methodology for your model and traffic.

If the hardware choice itself remains open, the AI accelerator guide explains GPU, NPU, FPGA, and ASIC tradeoffs, while the accelerator-versus-GPU inference comparison covers when specialized hardware can justify its narrower software path.

Common multi-GPU serving mistakes

• Treating combined VRAM as one pool. The runtime must shard weights and cache according to a supported strategy, with memory reserved on each rank.
• Increasing tensor-parallel size by default. More ranks add communication and may slow a model that already fits.
• Testing one prompt at concurrency one. That measures a latency floor, not production capacity.
• Mixing long and short traffic without observing the scheduler. Large prefills can delay short requests and distort tail latency.
• Ignoring rank failure. A sharded replica usually fails as a unit when one required worker disappears.
• Assuming a collective benchmark is an inference benchmark. It validates part of the path, not the full workload.
• Hard-coding a provider configuration. Recheck current GPU availability, topology, regional capacity, software support, and price before production.

Frequently asked questions

When should I move from one GPU to multiple GPUs?

Move when the complete serving memory budget does not fit safely on one GPU, or when a measured single-replica configuration cannot meet the workload’s latency and throughput objectives after reasonable tuning.

Is tensor parallelism always faster?

No. It lets a model use several GPUs, but each layer can require communication between ranks. It helps when sharding is necessary or benchmarks show a better result on the available topology.

Should I shard a model or add replicas?

Shard when one request needs several GPUs to run the model. Add replicas when a complete model fits and the goal is more independent request capacity, simpler isolation, or easier failure recovery.

Does multi-GPU serving fix long-context memory pressure?

It can provide more sharded capacity, but long contexts and concurrency still expand the KV cache. Quantization, realistic context limits, prompt design, cache management, and request routing remain part of the solution.

What should I measure after adding GPUs?

Measure TTFT, TPOT or inter-token latency, end-to-end latency, throughput, goodput, queue time, error rate, memory headroom, and communication time under the same request distribution used for the single-GPU baseline.

Can I span multiple hosts?

Yes, when the serving engine supports it and the workers share a consistent environment. Cross-host networking, security, model distribution, orchestration, and failure recovery become first-order concerns, so test them separately from a single-host result.