LLM fallback: queues, retries, and provider limits

LLM fallback is the use of an alternative execution path when the preferred path cannot serve a request. A useful fallback policy states which failures justify another attempt, which alternatives are acceptable, and when to return an error. Adding more destinations alone does not answer those questions.

An inference server reaches capacity while a user waits for an answer. You can wait for a slot, retry a failed forward, move to another server, or send the request to an external provider. Each choice changes how long the user waits and what the system is allowed to do.

This guide is for developers and platform engineers operating LLM applications, including chat products and AI agents. It uses Hivenet Router to explain queues, retry budgets, local fallback chains, and external provider fallback. If you are still choosing a routing approach, start with our guide to LLM routing. Here, the focus is what happens when the preferred backend cannot respond. Hivenet Router is an open-source project in public alpha; verify this behavior for the revision you deploy.

Evidence and scope: This is a policy-design and acceptance guide, based on Hivenet Router documentation at commit f97fec8, reviewed on November 11, 2026. The YAML fragments illustrate documented configuration fields; they are not a tested deployment. No failure tests, provider calls, recovery timings, or performance benchmarks are reported here. The examples assume cross-site redirects and model wake-from-zero handling are disabled.

LLM fallback, retries, and queues: what changes?

A queue holds work until capacity becomes available. A retry makes another forwarding attempt after a failure. A local fallback step changes which agent-backed servers are eligible. Provider fallback sends a supported request to a separate service, potentially using a different model.

In Hivenet Router, the primary policy runs first, followed by the local fallback chain in order. A configured external provider is the last resort for supported chat requests. An exhausted or failed chain returns an error. The fallback-chain documentation defines that order and the conditions that advance it.

“Local” means a backend connected through a Router agent. It does not mean the server is in the same building, country, or region. Geographic requirements must be expressed in the policy and supported by accurate agent metadata.

Failover routing strategies: which errors should trigger recovery?

A backend that is unavailable and a request that is invalid need different responses. Trying the same oversized prompt on every server serving the same model wastes capacity. Moving a valid request away from an unhealthy server can help.

HTTP status codes alone are insufficient for a failover rule. A backend overload response and a Router quota rejection can have different meanings even if both look like rate limiting to the client. Keep the error code, source, and request ID available for diagnosis. Backend overload or rate limits can consume a local retry budget, while Router token-quota rejection stops recovery. A failing external LLM provider can also return rate limits or a server error; Router returns its provider-error response after that one call. Check the layer reporting the error before deciding whether to retry.

Also inspect the application's own retry behavior. Client retries can create additional requests beyond the Router's per-request budget. Decide which layer owns recovery and cap total attempts and elapsed time. Retrying a generation does not guarantee an identical answer, and an application that executes model-selected actions needs its own protection against repeated actions.

Exponential backoff and circuit breakers in application code

If application code retries transient errors such as temporary network errors, use a bounded policy. AWS's guidance on retries and backoff explains why immediate retries can increase load on a struggling service. Exponential backoff increases the wait time between retries; jitter varies that delay so clients are less likely to retry together in a thundering herd. Cap the delay and attempt count within the application's total deadline. Authentication failures or invalid requests usually need correction rather than repeated API calls.

A circuit breaker temporarily stops calls to a persistently failing dependency and later permits limited probes. In an application using an external LLM API, a circuit breaker can prevent repeated calls to a failing provider while it recovers. It serves a different purpose from choosing a fallback model. These are general application design patterns, not settings implemented by the YAML fragment below. Review the retry logic in your SDK or AI gateway before adding another layer.

Load balancing distributes work among eligible backends during normal operation. Failover routing strategies define what happens when the preferred path cannot serve. They work together, but neither automatic failover nor a backup provider guarantees high availability if every path shares the same failing dependency.

Keep queueing inside the request deadline

Hivenet Router can wait when suitable agents exist but all are at their declared capacity. Its ordinary capacity queue is per model, with a default depth of 30 waiting requests. --queue-depth 0 disables queueing. Requests released by the separate hold-at-zero mechanism have additional behavior described in the docs, so the ordinary queue depth is not a universal limit on all waiting work.

A waiting request does not reserve a particular server. When capacity becomes available, Router evaluates the current step again. This lets a newly available agent serve the work, but it also means queueing cannot promise a specific destination.

The queue uses the original request deadline. Later local steps and external fallback do not receive a fresh time budget. If the Router request deadline has already expired, the processor returns a timeout instead of starting another local forward or provider call. As a hypothetical example, a request with a 60-second deadline that spends 55 seconds waiting has at most about five seconds left for everything that follows.

Use queueing when brief local saturation is acceptable for the workload. If reaching another eligible destination quickly matters more than waiting, evaluate disabling the capacity queue. Neither setting is universally better: test against the user-facing latency limit and the capacity available in the fallback pool.

Measure waiting separately from generation. A backend can generate tokens quickly after admission while users still experience a long delay before the first token. Our guide to TTFT and token throughput explains why those measurements answer different questions.

Design a local fallback chain with explicit boundaries

The following policy fragment prefers one tagged vLLM pool and then a second pool in the same region. It assumes both pools already register the exact model requested by the client and the required capability. Configure the agents so that the primary and secondary tags identify separate pools.

Illustrative policy fragment: The fields and nesting follow the versioned policy reference. This fragment does not install agents, configure authentication, or establish backend capacity. Its retry values illustrate a bounded budget and are not production defaults.

routing_policy:
  match:
    region: EU-France
    engine: vllm
    tags:
      - primary
  strategy: least-loaded
  max_tries: 2

fallback_chain:
  - name: france-secondary
    match:
      region: EU-France
      engine: vllm
      tags:
        - secondary
    strategy: least-loaded
    max_tries: 2

Each step repeats region: EU-France. A later step with match: {} would widen eligibility and could accept another location. Region matching is exact and case-sensitive: EU does not automatically include EU-France. Confirm registered values before relying on the filter.

The only implemented ranking strategy documented for these steps is least-loaded. Every local step retains the requested model and capability. A fallback agent registered under a different model name is not a replacement, even if you consider its model equivalent.

You can deliberately vary region, engine, or hardware in later steps, but evaluate the resulting behavior. Serving the same model through a different engine does not prove identical sampling, response fields, tool support, or streaming behavior. Use your application's actual requests in the compatibility tests.

Understand what max_tries counts

max_tries limits retryable forwarding failures within a step. It does not count time spent in the queue, rejected candidates, or losing a race for the last capacity slot. The first connection-level recovery attempt for an agent and request is also budget-free. Do not describe the field as an exact cap on all network activity.

After a failed forward, that agent is excluded for the rest of the current step. The failed-agent set resets when Router advances, so overlapping fallback steps can try the same agent again. Separate pools make the example easier to reason about; adding several steps with identical filters may only repeat the same failures.

Keep retry budgets small enough to leave time for useful alternatives. Each retry attempt can delay the next destination, so sequential failover adds latency when earlier attempts fail slowly. A long chain with generous budgets can spend the deadline proving that several unavailable backends are still unavailable.

External LLM fallback changes the provider and model

Hivenet Router supports one top-level fallback_provider using openai or anthropic. It runs after the local steps are exhausted for an eligible request. It is separate from fallback_chain, not another item inside that list.

Illustrative provider fragment: Add this top-level block only when the external destination is permitted for the workload. Replace the placeholder with a model available to the provider account; the placeholder is not a working model identifier. Provider credentials and application compatibility must be configured and checked separately.

fallback_provider:
  engine: openai
  model: "<provider-model>"

The original requested model, or primary model, is authorized before routing. The provider call then uses the model configured in fallback_provider. That substitution can change response quality, context limits, tokenization, and cost. It also changes who receives the request content. A local region filter does not constrain the external provider's processing location.

Provider credentials belong in the Router environment, using HIVENET_ROUTER_OPENAI_API_KEY or HIVENET_ROUTER_ANTHROPIC_API_KEY. Router loads them at startup. Restart it after adding or rotating a key; a policy reload does not reload credentials.

The provider-fallback documentation describes the supported fields and translations. The predictable starting point is a non-streaming text request to /v1/chat/completions. Embeddings and reranking do not use this provider fallback path. Do not rely on it for Anthropic token counting or complete passthrough of Anthropic-native Messages fields.

Check the fallback model, fields, and streaming

The adapters reconstruct a supported chat request rather than forwarding the original JSON unchanged. OpenAI fallback may omit fields such as tools, tool choice, structured-output settings, stop sequences, or log probabilities. Anthropic translation is text-focused and does not fully preserve images, audio, tools, tool results, or provider-specific fields. A tool-only Anthropic response cannot succeed through the current translator because it requires a text block.

Provider fallback is not implemented as a streaming adapter. Set stream: false for the supported use case. Do not promise that an interrupted stream can resume on another provider or that the adapter will seamlessly convert a streaming request into a complete response. OpenAI server-sent events may fail decoding where the adapter expects JSON.

An OpenAI-compatible API still needs application-level compatibility checks. A successful simple prompt does not establish that your tool calls, structured outputs, or multimodal requests survive a fallback.

For local streaming responses, an initial HTTP 200 does not prove that generation completed. The error reference documents a terminal stream_truncated frame when an upstream stream ends early. Treat that frame, or a stream ending without its normal completion marker, as partial output. Retry only when repeating the operation is safe. Local routing counters alone do not prove successful stream completion.

Keep provider spending and timeouts visible

The provider client's documented HTTP timeout is 120 seconds, but the remaining original request deadline also applies. The effective allowance is the shorter of those limits. A nearly exhausted local deadline cannot be repaired by the longer provider timeout.

Router makes one outbound provider call. It does not automatically retry that provider, try a second provider, or return to local routing after failure. The documented provider-error response is HTTP 502 with backend_error; test deadline expiry and client cancellation separately in your deployment.

Local token usage accounting is not a provider spending cap. Request-rate deductions, input-token rate charges, and daily input reservations taken before routing remain in place. Global and per-key occupancy reservations are released when routing hands the request to the provider. Provider output does not charge the local output-token rate bucket or trigger the normal post-response daily output-token deduction. Provider usage is recorded for observability, with attribution to the original requested model. Use provider-side spending controls and alerts alongside Router metrics.

Sometimes an error is the correct outcome. If the remaining destination cannot preserve required data handling or application behavior, leave it out of the chain and make the failure useful to the caller.

Observe successful recovery and failed attempts

Track the primary path, local fallbacks, provider fallbacks, and exhausted chains together. Compare end-to-end latency, failure rates, token usage, and completed responses; routing counters alone cannot tell you whether failed requests recovered. A rise in fallback traffic can indicate insufficient primary capacity, an unhealthy backend, restrictive gates, or incorrect metadata. It can also conceal a growing external-provider bill while the application's overall success rate appears stable.

The Prometheus metrics reference documents the available series. Start with these counters, then use logs for individual request diagnosis:

Use outbound http_client_request_duration_seconds_* metrics and Router logs for provider failures as well as successes. A network failure appears with status_code="0". Use named-step logs to distinguish which local fallback served or exhausted a request.

Send a valid UUID as the client correlation value in X-Client-Request-ID; Router returns its own identifier in X-Request-ID. Keep both with the application outcome. Correlation does not make a generation idempotent: reusing an ID does not prevent a new inference operation. A successful provider response includes processed_by: "provider:openai" or processed_by: "provider:anthropic", which helps verify that the intended provider path was used.

Plan graceful degradation when every LLM path fails

A layered recovery policy combines bounded retries for transient failures, local failover, an approved external destination, and an application response when none succeeds. A single provider can remain a point of failure, but using multiple providers does not guarantee recovery if they share an unavailable dependency or all lack capacity. Avoid treating a longer fallback chain as evidence of higher availability.

Graceful degradation belongs in the application code. For example, a support assistant could offer a verified help article or a human support path during total outages. A drafting tool could preserve the user's input and explain that generation is temporarily unavailable. These are design options to implement and test, not automatic Router responses. Make the reduced capability visible so users can distinguish a fallback response from a fresh model answer. Microsoft's circuit-breaker guidance includes returning a default response or invoking alternative functionality when a dependency is unavailable.

LLM fallback questions

Should you retry the same request or choose a fallback model?

Retry only when the error is transient, repetition is safe, and the deadline leaves room for another attempt. Persistent unavailability calls for an eligible alternative or a final error. The same prompt can produce a different answer on retry, even with the same primary model. In Router, local failover preserves the requested model; only the external provider configuration substitutes a fallback model. Bad requests and context-length errors stop the chain.

Can a fallback model come from the same provider?

That depends on the system implementing the policy. Router's local alternatives are agent-backed servers, and its external configuration selects one OpenAI or Anthropic provider and one fallback model. It does not define a primary provider followed by multiple backup providers. If that external provider fails, Router does not automatically try the same provider again or switch to another LLM provider.

Does a cheaper fallback model reduce costs?

Do not assume savings. The primary model may already have consumed work before recovery, and the fallback model can have different tokenization, output length, and billing. Selecting a cheaper model for simple queries is a separate routing decision; this Router policy does not classify task complexity or answer quality. Evaluate the fallback model on representative requests and reconcile provider charges with observed token usage before claiming a cost improvement.

Acceptance checks before sending real traffic

Use an isolated environment with known agent metadata, fixed non-sensitive payloads, and an authorized provider test account if provider fallback is in scope. Record the Router and agent revision, backend and model versions, complete policy, credentials' scope, and timeout settings without exposing secret values. Disable application retries for the first run so Router behavior can be observed separately.

Define the acceptable client deadline, error behavior, output requirements, data destinations, and spending limits before the run. The table below is an acceptance plan, not a report of completed tests. A case passes only when its evidence supports the stated criterion; record failed and not-run cases as well.

Keep the test window and request-ID mapping with the results. Combine client responses, step logs, local counters, outbound provider activity, and provider billing or usage records where available. That makes successful recovery distinguishable from a second uncontrolled request or a response that arrived too late.

After a controlled test, restore the intended policy and verify the normal primary path. Do not test total outages by disabling a shared production pool or sending sensitive payloads to a provider.

Then compare recovery behavior with the application's requirements: acceptable delay, output format, data destination, and cost. The wider production inference guide connects these checks to capacity planning and operations.

Start with a short local chain whose behavior you can explain from the logs. Add external fallback only after testing its model and request compatibility. The Hivenet Router quickstart provides the deployment starting point; the policy should reflect the destinations your application can actually use.