LLM routing explained: model selection, gateways, and backend routing

A large language model (LLM) router directs an application's inference requests to a model, a provider, or a serving backend according to a routing policy. What it selects depends on the system: some routers choose between models, while others distribute requests across servers running the same model.

That distinction matters when an application grows. A prototype might call one model server directly. A production deployment may need several replicas, different models for different tasks, access controls, and a defined response when capacity runs out. These requirements can share an endpoint without being the same routing problem.

This guide explains the three decisions behind LLM routing, how common routing strategies work, and when a self-hosted routing layer is useful. It also shows where Hivenet Router fits, using an illustrative two-server example.

What does an LLM router choose?

Start with the decision the application needs to make. Does it need a different model, a different provider, or another server that can run the same request? Product names such as AI gateway, LLM gateway, and inference router often overlap. The useful comparison is what each system can select and which signals it uses.

One product may perform more than one of these jobs. For example, LiteLLM's routing documentation covers deployment load balancing alongside retries and fallbacks. A gateway is therefore not necessarily limited to translating API requests, and a router does not necessarily choose a different model.

An application can also make these routing decisions itself. If a developer explicitly requests a model, a backend router may keep that choice fixed and select only an eligible server. Adding a routing layer does not automatically mean handing model selection to another AI system.

Model routing: rules, semantic routing, and evaluation

Model routing is useful when multiple models serve different needs. A small model may meet the quality threshold for extracting a few fields from short documents. Another may be more suitable for long-context analysis or a particular coding task. The routing rules decide which requests go where.

Static routing uses predefined rules. An application might send every request from its summarization feature to one model, or select a model according to an explicit task label. These rules are relatively easy to inspect, but someone must maintain them when the application or model catalog changes.

Dynamic routing changes a decision using information available at request time. That information might be estimated task complexity, current availability, or a budget limit. The term alone does not tell you whether a machine-learning model makes the decision. A rule that reacts to a current signal is dynamic too.

Semantic routing uses information about the request's meaning or task to help select a destination. Implementations can use embeddings, classifiers, or learned routing policies. Research projects such as LLMRouter explore several model-selection approaches. Before adopting one, check what it actually evaluates and how its decisions were tested.

The practical question is whether the selected model produces an acceptable answer on your workload. Test representative requests, including difficult and uncommon cases. Compare response quality, latency, and total cost against a simpler baseline, such as always using a single model. Include the router's own processing cost and any extra calls made after a poor answer.

Record model parameters, context limits, quantization, and the serving configuration alongside each result. Compare model performance using the same task set, and check each required model capability independently. This helps you choose a suitable model without treating a family name or size label as proof of quality.

Cheaper models can reduce the price of an individual call. They do not guarantee a cheaper successful task. If the application repeatedly retries, escalates to another model, or needs human correction, the result can cost more. A routing strategy should earn its place through evaluation rather than a promised percentage saving.

API gateways: one interface, several destinations

An AI gateway can act as a proxy server, giving an application a common way to reach deployments from several model providers. Depending on the implementation, it may manage provider credentials, authentication, usage reporting, rate limits, cost controls, and fallback policies. This can reduce the amount of provider-specific routing logic spread through application code.

A unified API still needs a compatibility check. Two endpoints can accept similar chat requests while differing in tool calling, structured output, streaming, context limits, or error behavior. Switching a base URL is not sufficient evidence that the application will behave the same way.

Keep an explicit list of the features the application uses and test each destination against it. Our guide to OpenAI-compatible APIs explains the checks that matter when comparing endpoints.

Access controls also have separate scopes. A client API key identifies the application or user calling the gateway; provider credentials let the gateway contact an upstream service. Quotas and spending limits need to be enforced at the appropriate boundary. Centralizing credentials does not remove the need to control access to the gateway itself.

Provider fallback introduces another decision: whether a request may leave its original environment. Review the destination's capabilities and data handling before enabling that path. A response from another provider can also come from a different model, so successful delivery does not establish equivalent output.

Inference routing and LLM load balancing

Backend routing becomes useful when several inference servers can serve the application. For self-hosted models, a backend might be a vLLM deployment running on a GPU instance, another supported model server, or a replica elsewhere in the same permitted environment.

The router first needs to establish which backends are eligible. A reachable server is not enough: it must serve the requested model and required capability, satisfy the routing policy, and have room for the request. Only then does the load-balancing strategy choose among candidates.

LLM requests make this more complicated than counting identical jobs. A short classification request and a long generated answer occupy capacity for different lengths of time. Ongoing streams, queued work, and the model server's own scheduling all affect the experience. Request counts are useful operational signals, but they do not describe every part of the workload.

Likewise, filtering and ranking are different operations. A policy might exclude a server whose available metric crosses a threshold, then choose the least-loaded server from those remaining. That does not mean the router selects the lowest-latency server or the server with the best cache match.

An illustrative example with two model servers

Suppose two healthy servers run the same model and both pass the request's capability and policy checks. Server A has two active requests and a declared capacity of four. Server B has three active requests and a declared capacity of eight.

A strategy based on active requests divided by declared capacity compares 2 ÷ 4, or 50%, with 3 ÷ 8, or 37.5%. It selects Server B, even though B has more active requests. This is the least-loaded calculation documented for Hivenet Router's backend selection.

The numbers are illustrative, not benchmark results. The example shows how the decision is made; it does not predict response time or establish that eight concurrent requests is a suitable capacity for a particular server. Operators need to set capacity from testing with their models, hardware, and request mix.

The requested model has not changed. The router has selected where the request runs. A separate model-selection policy would be needed to choose another model because it is cheaper or more suitable for the task.

Queues, retries, and fallback solve different problems

A wait queue holds a request until capacity becomes available. It can absorb a short burst, but waiting consumes the user's latency budget. A queue that grows without an effective limit can turn overload into slow failure.

A retry makes another attempt after a failure. Retry budgets and timeouts need to fit within the application's overall deadline. Retrying every error is wasteful: an invalid parameter or an oversized context will not necessarily become valid on another attempt. Repeated attempts can also add load when a service is already struggling.

Fallback moves to a configured alternative when the preferred path cannot serve the request. A local fallback might relax a placement preference while retaining the same model and capability. An external-provider fallback may change both the destination and the model. Define which constraints must survive every step.

Streaming requires particular care. Once part of an answer has reached the client, restarting the request elsewhere is not a transparent continuation of that answer. The client needs defined behavior for interrupted streams, and any downstream action triggered by generated output needs its own safeguards against duplication.

Test failure behavior deliberately: a full backend, an unavailable server, an invalid request, a timeout, and an interrupted stream. These cases reveal whether the application fails clearly and within its deadline. Healthy-request throughput alone cannot answer that question.

Where Hivenet Router fits

Hivenet Router is open-source software for routing requests to inference backends you operate. It sits in front of model servers; those servers execute the models. You can run it on infrastructure of your choice without requiring a Hivenet account or Hivenet infrastructure.

Agents connect supported backends to the Router. Its documented integrations include vLLM, SGLang, Ollama, llama.cpp, and Infinity, with supported request types depending on the backend. The Router introduction describes the integration scope.

For backend selection, Hivenet Router checks the requested model, health, capability, and capacity, applies configured metadata filters and available metric gates, then ranks eligible agents using least-loaded. A metric gate can be skipped when that metric is unavailable, so operators must verify what their chosen backends report.

Hivenet Router's documented local fallback retains the requested model and capability. Its configured OpenAI or Anthropic provider fallback is limited to supported non-streaming chat requests. It should not be treated as an interchangeable backup for every API feature or workload. Review the provider-fallback limitations before enabling it.

Current main-branch documentation also describes configured model aliases, such as auto, that resolve to a concrete model through structural and keyword rules. That selection step is separate from backend ranking and does not use an LLM to interpret arbitrary prompt meaning. See the model-alias documentation for its scope.

The Router behavior described here was checked against the project's documentation on November 11, 2026. The two-server calculation is illustrative; no deployment benchmark is claimed.

The project is in public alpha. Check the documentation for the version you deploy: the repository's main branch can describe changes beyond a tagged release. In particular, do not infer learned prompt-intent routing or a lowest-latency ranking strategy from the general term “LLM router.”

Hivenet Inference Service is a separate managed offering in which Hivenet operates the serving infrastructure. Running the open-source Router leaves deployment and operations with you. The two routes suit different responsibilities, and the choice belongs in the wider production inference architecture.

How to choose an LLM router

Write down the routing decision you need before comparing products. An application that needs multiple large language models evaluated by task has different requirements from a team that needs to distribute incoming requests across replicas. Separate those requirements even when a single product supports both.

Then check the operating model. Managed routers can transfer some deployment work to a provider, subject to the service's actual scope. Self-hosting gives your team control of configuration and deployment, along with responsibility for availability, updates, and operational overhead. An open-source license does not pay for the servers or the people maintaining them.

Look at how policies can be inspected and tested. If you need custom routing logic, determine whether configuration is sufficient or whether you must maintain code. Check that fallback logic preserves required model capabilities and destination restrictions. Automatic failover is useful only when the alternative remains acceptable for the application.

For cost tracking, identify what the router measures and what the bill includes. Token usage may explain provider charges, while a dedicated GPU deployment can keep accruing infrastructure costs when idle. Include unsuccessful attempts and any separate model-selection calls. Cost optimization depends on that complete picture.

Finally, replay representative requests before moving production traffic. Keep a record of the policy and model versions used, the selected destinations, and the results. This gives future changes a baseline and makes routing decisions easier to investigate.

What to measure before relying on a router

Start with the application outcome: acceptable answers delivered within the required time and budget. Then inspect the serving path to understand where that outcome improves or deteriorates.

Measure time to first token, generation speed, end-to-end latency, and error rates across a representative workload. Look at latency percentiles as well as averages, because a satisfactory average can hide slow requests. Our guide to LLM inference metrics explains how these measurements relate to the user experience.

Add routing-specific visibility: queue waits, selected backends, exclusions, retries, fallback frequency, and rejected requests. If you change models, measure answer quality by task type too. An apparent infrastructure improvement can conceal worse answers or more escalation calls.

The router itself becomes part of the service you operate. Protect its administrative interfaces, configure client and agent authentication, plan updates and recovery, and set a deliberate policy for logs. Prompt logging can expose sensitive content; self-hosting one component does not establish the data handling of the whole application.

Use the simplest routing arrangement that meets the requirement. One healthy server serving one model may be adequate for an early deployment. A shared routing layer becomes more useful when applications need consistent access controls or several eligible backends, and maintaining those decisions separately in each client becomes difficult.

Common questions about LLM routing

Does an LLM router always select the best model?

No. Some routers choose among models; others route an explicitly named model to a suitable backend. Even a model-selection system needs evaluation against a defined quality target. “Best” depends on the task and the acceptable tradeoffs.

Does load balancing add GPU capacity?

Load balancing distributes requests across available capacity. It does not, by itself, provision GPUs, start model servers, or increase their throughput. Autoscaling requires an additional mechanism that changes capacity.

Can a router keep every request on self-hosted infrastructure?

That depends on the complete configuration. Restrict eligible destinations and review every fallback path, along with logging and monitoring. An enabled external-provider fallback can send request content outside the self-hosted deployment.

If you already operate model servers and need a shared routing layer, start with the Hivenet Router quickstart. Use a test deployment to verify request compatibility and failure behavior, and inspect the open-source repository for the version you intend to run.