vLLM load balancing: architecture and validation

vLLM load balancing distributes inference requests across model-serving replicas. With Hivenet Router, an application calls one endpoint while an agent beside each vLLM server connects it to the routing layer. The Router selects an eligible backend; vLLM executes the model.

This guide is for platform engineers and developers planning self-hosted inference. It explains a two-replica architecture, how Hivenet Router selects agents, and how to evaluate request distribution, overload, backend loss, and recovery. Those checks help distinguish useful capacity from a deployment that merely answers a health check.

Evidence and scope: Router behavior below follows the versioned documentation at commit f97fec8, reviewed on November 11, 2026. Hivenet Router is in public alpha. This is an architecture and validation guide; it reports no executed two-GPU deployment, failover timing, or performance benchmark. The checks below are an acceptance plan for a chosen deployment, not recorded test results.

If you are deciding whether you need model selection, a gateway, or backend routing, start with our guide to LLM routing. Here, both replicas serve the same model. You operate the routing layer and inference servers, including the reliability and operational overhead of the Router itself.

Core architecture: one load balancer and two vLLM replicas

The reference topology has one Router host and two GPU hosts. Each GPU host runs an independent vLLM server and its own Router agent. Applications call the Router HTTP API; agents initiate their connections to the Router and forward selected requests to their local inference backend.

Both servers load model weights independently. This does not combine the memory of two machines into one larger model. Tensor parallelism within a vLLM deployment is a separate choice. Two backend replicas also leave a single Router process as a potential failure point.

The versioned bare-metal quickstart documents HTTP on port 8080, agent authentication on 50051, the data plane on 9000, and metrics on 2112. Compose can publish different ports. Keep one deployment path and document which addresses the client, agents, and monitoring system can reach.

• Limit the client API to intended clients and provide HTTPS at the client boundary.
• Allow agents to reach the Router authentication and data-plane ports. A connection to port 50051 alone does not prove that request forwarding works.
• Keep administrator and metrics endpoints restricted to operators and monitoring systems.
• When an agent and vLLM run directly on the same host, vLLM can bind to loopback. Containers have their own networking rules; verify what the agent's backend address actually reaches.

Use the vLLM agent reference for deployment commands. Pin the Router and agent revision together, and choose a compatible vLLM release for the hardware. For source builds, check the selected revision's go.mod: this reviewed revision declares Go 1.27.3, while its quickstart lists an older prerequisite. That discrepancy is another reason to verify the build environment rather than treat a documentation example as a tested release combination.

Why prompt length and KV cache matter for vLLM load balancing

LLM inference has two main phases: prefill processes the input prompt, and decode generates output tokens. Long prompts increase prefill work; long answers keep decoding active for longer. Equal request counts can therefore represent different amounts of work. A routing policy must be evaluated against the input and output lengths your application actually sends.

vLLM’s PagedAttention design organizes key and value cache data into blocks. This manages memory inside an inference engine. It does not select a replica or share cache state between the two independent servers in this topology.

With automatic prefix caching, an engine can reuse computed KV state for a matching prompt prefix. That saves repeated prefill computation; it does not eliminate the work of generating new output tokens. Routing related prompts to a replica with reusable state can be useful, but cache locality and current load can pull the decision in different directions.

Independent replicas versus a vLLM data-parallel deployment

vLLM also has internal, hybrid, and external data-parallel deployment modes. Internal balancing uses running and waiting queues to distribute work across DP ranks behind an API endpoint. In hybrid mode, an upstream load balancer selects a node, and that node’s API server balances across its local engines. External balancing selects endpoints outside that internal scheduler. A DP rank is a data-parallel worker rank, not a Hivenet agent ID.

This guide considers two independent vLLM servers with an external Hivenet routing layer. It does not configure vLLM’s coordinated data-parallel modes or split prefill and decode across separate worker pools.

Establish model identity, authentication, and capacity

Make the replicas comparable

Record the vLLM version, model and tokenizer revisions, chat template, quantization, GPU model and memory, drivers, context limit, and serving options for both replicas. A model family name alone does not establish equivalent behavior. Start with matching settings so that unexpected differences have fewer possible causes.

Check each backend's /health and /v1/models endpoints, then send a small generation request directly to each server. The documented readiness response is HTTP 200, but a successful health check is weaker evidence than a valid generated response. Confirm the exact served-model identifier before configuring the agents and client requests.

Each Hivenet agent registers one model. Set --model explicitly when selecting a particular model; automatic discovery otherwise uses the first model returned by the backend. Give the agents distinct persistent identity files and machine labels, then record the peer-ID-to-host mapping. Do not copy one agent's private identity to another host.

Keep the authentication boundaries separate

The authentication reference distinguishes agents joining the network, clients calling /v1/*, and operators calling /admin/*. Agents share the configured JWT secret. Client credentials and administrator credentials serve different purposes, and client authentication must be configured for a shared deployment. A Router client key does not automatically authenticate the agent to a protected vLLM backend.

Use your normal secret-management process and test these boundaries separately. Rejecting a missing or invalid key on the protected API is a different check from confirming a valid client's permitted model access. Enabling an API key also does not encrypt a plain HTTP client connection.

Set concurrency from measured backend capacity

Hivenet's agent --capacity limits concurrent work assigned by the Router. It does not configure vLLM's scheduler. The vLLM CLI reference defines --max-num-seqs as the maximum sequences processed in one iteration and --max-model-len as the combined prompt-and-output context length. Matching two numeric settings does not prove that the GPU can sustain the resulting traffic.

Model weights, KV cache, runtime allocations, prompt lengths, and output lengths affect the usable limit. Begin with a controlled workload, observe memory and queue behavior, and increase concurrency only while the application's latency and error requirements remain satisfied. There is no universal concurrency value established by this guide.

How Hivenet Router chooses a vLLM backend

The versioned routing reference describes model, health, capability, and capacity checks before an eligible agent is selected. Static policy filters and optional metric gates can narrow the candidate set further. The default policy applies no optional metric gates and uses least-loaded ranking: active requests divided by declared capacity.

For an arithmetic example, an agent with two active requests and capacity eight has a load ratio of 0.25; one with two active requests and capacity four has a ratio of 0.5. The first ranks ahead if both pass the other checks. These are illustrative values, not measured capacity or a hardware recommendation.

Equal-load ties use deterministic peer-ID ordering. A sequence of isolated requests can therefore reach the same worker repeatedly. This is not round-robin scheduling, and equal request counts are not the acceptance criterion. Verify overlapping work before judging distribution.

When matching healthy agents are full, the Router can wait in a per-model queue. Queue behavior, request deadlines, retry budgets, and any fallback policy affect what the client observes. Record them in the test configuration; otherwise, an apparently successful request might conceal a long wait or additional backend attempts.

Validate request distribution before increasing traffic

Run these checks in an isolated test deployment with a fixed, non-sensitive request payload. Choose an application deadline and acceptable error behavior before the run. Keep client retries disabled for the initial functional test, and record the Router's own retry settings so that client completions are not confused with backend attempts.

For the concurrent batch, preserve each response body, HTTP status, transport error, and elapsed time separately. Begin below the configured combined capacity. If short requests finish before they overlap, repeat a controlled batch or change the output length while watching active work. Do not infer that one backend is unused merely because sequential calls selected the same peer.

The versioned metrics reference documents hivenet_agent_requests_success_total with peer-level labels. Compare counter increments for the two recorded identities and correlate them with backend logs. A missing series before the first request can be normal. During retries, backend attempts and client requests can differ, so one aggregate counter is not a complete request audit.

The broader hivenet_routing_requests_routed_total counter has region, engine, model, and tenant labels. If both agents share those values, it cannot distinguish the two servers. Per-agent lifetime counters can be restored from BadgerDB when an agent registers again; do not assume a Router restart resets them. Changed identities or deleted persistent state also invalidate a simple before-and-after comparison.

A successful non-streaming Chat Completions request establishes only that tested path. Tool calls, structured output, streaming, and the application's other required API features need separate cases. The OpenAI-compatible API buyer guide explains how to scope that acceptance work.

Test backend loss, recovery, and interrupted streams separately

Backend failure and connection failure are different events. The routing reference describes a bounded connection-recovery attempt for a stale Router-agent connection; the agent's reconnect loop restores its outbound connection. That behavior is not evidence that a failed vLLM process will recover, or that an interrupted generation can continue elsewhere.

For a backend-loss test, first finish the normal batch. Stop only the dedicated vLLM process on one test host, leaving its agent and the other backend running. Record the stop time, health-state changes, request outcomes during the transition, and outcomes after the stopped backend is recognized as unavailable. A request sent during detection can behave differently from one sent after health state settles.

The acceptance question is whether new requests use the remaining eligible capacity within the defined deadline and error budget. Record failures as well as successes. Do not turn an observed detection time into a general failover guarantee.

Restart the same backend configuration, verify local readiness and generation, then inspect agent registration and eligibility before repeating the concurrent batch. Require observed successful work from the recovered replica. Record whether the agent reconnected or registered again; a healthy badge alone does not demonstrate a recovered inference path.

Test agent-process loss and network interruption separately if those events are part of the deployment's reliability requirements. They exercise different boundaries from stopping only the backend. Also test the Router process itself: two vLLM replicas do not establish high availability for one routing process.

For streaming, record time to first output, chunks received, completion or error signaling, and client cancellation. Interrupt a test stream deliberately and check what the application displays. Already-delivered output cannot simply be replaced by a second response. Do not claim transparent stream resumption or exactly-once generation from a non-streaming recovery test.

The agent reference states that a routed stream retains its capacity slot until it drains. Verify slot release after normal completion and cancellation in the selected version. If active-request counts or backend work remain unexpectedly elevated, investigate before increasing concurrency.

Usage-based routing, latency-based routing, and cache locality

“Usage-based routing” can mean different things across products: request occupancy, token budgets, or provider quotas. In this setup, the default least-loaded strategy uses active requests relative to declared capacity. It is not latency-based routing: a backend's historical response times do not determine its ranking. For bursty workloads, examine both the Router's admission decisions and the inference layer's running and waiting requests. A server can be eligible while still producing response times that your users find unacceptable.

Requests from the same session are not guaranteed to reach the same backend. Subsequent requests can therefore land on a different replica, even when the conversation shares a long prompt prefix. Do not assume session affinity, consistent hashing, or cache-aware routing from this setup. Cache behavior needs its own measurement and an explicitly supported routing strategy.

Load balancing also does not supply an orchestration layer that starts new GPU workers when traffic rises. This topology routes between already-running processes. To scale beyond them, you need a separate capacity-management mechanism and tests for workers joining, draining, and leaving the system.

How the separate vLLM Router project differs

The independently maintained vLLM Router project documents several policies, plus optional Kubernetes worker discovery. Its load-balancing reference describes these choices:

• Round robin: cycles across healthy workers without ranking their current load.
• Power of two choices: samples two healthy workers and selects the one with lower load.
• Consistent hashing: uses a session or user identifier to favor the same worker.
• Cache-aware routing: uses approximate prefix matches while accounting for load imbalance.
• KV-aware routing: uses KV reuse information and worker load, falling back to load when usable KV information is unavailable.

That project also supports separate routing policies for prefill and decode worker pools. These are alternative deployment capabilities, not features enabled by the Hivenet topology described here. Pin and test the chosen project’s version; its algorithm names and configuration are not interchangeable with Hivenet Router.

Rate limits, retries, and circuit breakers for production traffic

A small concurrent batch checks a functional path. Before sending production traffic, define the application's overall deadline and retry budget. If the client, proxy, and Router all retry independently, the number of backend attempts can grow unexpectedly. Use bounded retries and, where appropriate for the application, exponential backoff for transient failures. Avoid retrying invalid requests without correcting them.

Review any circuit breakers in the surrounding platform separately from Router health checks. A failure threshold, cooldown, and recovery probe are specific behaviors to configure and test, not features established by placing a load balancer in front of a server. During an outage, excessive retries can contribute to cascading failures by overloading the remaining capacity.

Separate rate limits from concurrency limits. A request-rate budget limits arrivals over time; agent capacity limits simultaneous work assigned by the Router. Neither proves a latency target will be met. Record rejected requests and queue waits as well as successful completions when evaluating the user experience.

For cost optimization, compare the cost of the complete deployment with useful output delivered within the required latency. Two rented replicas may both incur charges while idle. Keeping GPUs busy is not enough if requests fail, wait too long, or produce answers the application cannot use.

Monitor the inference layer alongside Router decisions

For each vLLM replica, compare running requests, waiting requests, KV-cache occupancy, prefix-cache reuse, time to first token, and completion latency. The vLLM production metrics reference documents vllm:num_requests_running, vllm:num_requests_waiting, and vllm:kv_cache_usage_perc, among other signals. Verify names and availability against your pinned release.

Correlate these with GPU utilization, Router active requests, per-agent success counters, queue waits, and errors. An uneven request count can be appropriate when jobs differ in length. A growing backend queue with worsening client latency is more useful evidence of overload than request distribution alone. Choose a sampling interval that captures your workload’s bursts without imposing excessive monitoring overhead.

Hivenet Router’s routing concepts reference separates metric gates from ranking. Optional gates can exclude agents based on reported queue, cache, or latency signals, but the default policy adds no optional metric gates and still ranks by active requests divided by capacity. Missing metrics skip their gates. Reading a metric on a dashboard does not mean the routing policy uses it.

Troubleshoot the observation before changing the policy

Record results before production use

Keep one evidence record containing the Router and agent revision, vLLM version, model/tokenizer revisions, GPU and driver details, serving options, policy, deadlines, retry settings, and request payload. Add health snapshots, peer-ID mappings, response statuses, counter deltas, and the failure/recovery timeline. Mark each test passed, failed, or not run, and redact credentials and sensitive request content.

Functional distribution is not a throughput benchmark. Compare the direct-backend baseline and routed path under the same representative workload, including prompt and output lengths, arrival rate, concurrency, and warm-up conditions. Measure time to first token, end-to-end latency, useful throughput, and errors. Account for Router queues and vLLM scheduling separately. Our inference metrics guide explains these measurements.

Before production traffic, validate required API-key restrictions, rate or usage limits, HTTPS, administrator access, client cancellation, and bounded retries. Establish rollback and capacity limits from those observations. Keep external-provider fallback out of the initial two-replica test because it can change the model and data destination.

After the test, stop only the lab processes and handle temporary credentials and response files according to your retention policy. If GPU instances were rented, stop or terminate the lab resources through the provider; stopping the Router alone does not end their charges.

Hivenet Router can connect inference backends on infrastructure you operate. Use the versioned configuration references to build the selected topology, then use the acceptance checks above to establish what that deployment actually does before relying on it.