← Blog
An orange line passes through pale outlined blocks toward a solid orange block, suggesting a model becoming active.
Published on
2026-11-18

LLM cold starts: scale-to-zero inference with Hivenet Router

An LLM cold start is the wait before a model that is not ready can serve a request. In scale-to-zero LLM inference, the model has no running replicas during an idle period. The next request may need to wait for resources, model loading, and backend initialization before generation can begin.

This guide is for developers and platform engineers running self-hosted inference who need to handle that first request. Hivenet Router can hold an eligible request within a deadline and expose pending demand for an external autoscaler. Router does not start or stop models itself.

You will configure the placement map and hold limits, connect a KEDA wake-up signal, and check client responses, timeouts, and recovery. The examples also explain when keeping a model warm may better fit the application's latency requirements.

Validation scope: Configuration and request behavior were checked on November 18, 2026 against Hivenet Router v0.1.7, the current Router documentation, and KEDA 2.21 documentation. The JavaScript response checker passed 20 offline checks on Node.js 24.22.3 using synthetic responses. No Router, Kubernetes, KEDA, or GPU deployment was run. The configuration below is an example to validate, not a recorded deployment result or a cold-start benchmark.

LLM cold start vs warm inference: what changes?

A cold request must wait for the model to become ready before it can use the normal serving path. A warm request reaches a model that is already ready, although it can still wait for admission or backend capacity. Keep those delays separate when diagnosing slow responses: scaling from zero, queueing, and generation call for different checks.

A cold start can include scheduling a pod, obtaining GPU capacity, pulling an image, reading model weights, and initializing the serving engine. Measure the complete path in your environment. Having a model available in storage does not establish how quickly it can serve a request.

Scale-to-zero is worth evaluating when idle periods are long enough to justify the extra startup work and users can tolerate the delay. If a first response must arrive within a tight deadline, keeping a ready replica may be the more suitable choice. Compare billable idle capacity, startup work, persistent storage, and the systems that remain running. Zero model replicas does not mean zero infrastructure cost.

Keep cold and warm requests separate in your results. The existing inference metrics guide covers latency measurement; this article focuses on what happens before a cold model can serve. For the wider deployment decision, use the production inference guide.

How inference engines load model weights into GPU memory

Large language models must load model weights before they can serve inference requests. NVIDIA’s model-loading explanation describes reading weights from storage into CPU memory and transferring them into GPU memory. Inference engines also have their own initialization work, so model loading time and total readiness time are different measurements.

For your deployment, record the model file format, storage location, network path, and loader alongside the GPU hardware. Storage bandwidth and transfers into GPU memory can limit loading speed. A model artifact in local storage does not establish that the inference engine is ready. If evaluating a different loader or a warm cache, compare the same model and hardware and report the cache state. The cited loader benchmarks describe a specific setup; they are not cold start latency results for Hivenet Router.

That NVIDIA article describes Model Streamer reading weights concurrently and overlapping storage reads with transfers to GPU memory. This is a backend loading technique to investigate if loading is the bottleneck. Check compatibility with your specific model, file format, and inference engine before changing loaders, then measure total readiness as well as loading time. Router’s request hold is independent of that choice.

How Hivenet Router handles scale-to-zero LLM inference

The Router v0.1.7 hold-at-zero reference describes four responsibilities:

  1. The application sends an authenticated request for an allowed model and keeps an overall deadline.
  2. Router checks placement and eligibility, holds the request within configured bounds, and reports pending demand.
  3. An external autoscaler reads that demand and starts the model workload.
  4. The agent registers when its backend is ready. Router releases held requests into normal admission and routing.

A healthy local agent takes the normal serving path. A suitable sibling site can take precedence through a redirect. The example below uses one site to isolate the local wake-up flow. It does not add another provider or change the requested model.

Holding a request is separate from retrying a failed call, waiting for an already-running backend's capacity, or falling back to a provider. Those paths belong in the existing queues, retries, and fallback guide.

Prepare the site, credentials, and model workload

Start with an authenticated Router deployment, a model backend that works while running, and an agent that advertises the intended model ID. The example calls that ID example-chat; replace it consistently with your actual served model. The Kubernetes Deployment name can differ from the public model ID, although this example uses the same name.

Hold at zero requires the cross-site redirect machinery and a sibling map that places the model on this site with scale_from_zero: true. This remains a requirement even when the example map contains only one site. Router needs a site ID, an allowed HTTPS domain suffix, and a protected redirect-signing secret file in the documented id:secret format. Each secret must contain at least 32 bytes.

Use dynamic client keys with administrator authentication. The release-pinned redirect reference says startup rejects this feature with static client keys from auth.yaml. With no auth configuration file, HIVENET_ROUTER_AUTH_MODE=dynamic selects dynamic mode and requires HIVENET_ROUTER_ADMIN_API_KEYS. An auth file takes precedence over that environment setting. Provision a client key restricted to the intended model through the documented dynamic-key workflow; do not use the admin key for inference.

Keep existing agent authentication, TLS termination, private administration access, and key storage in place. The Router security guide covers those boundaries. This article does not replace that setup or configure an unauthenticated shortcut.

Add bounded hold settings

The following are additional flags for an existing Router service command, not a complete deployment command. The paths and domain are examples. The hold limits are deliberately small trial settings, not sizing recommendations:

--site-id site-fr-1 \
  --redirect-secret-file /run/secrets/router-redirect-secrets \
  --redirect-allowed-domains .example.com \
  --wake-hold-budget 240s \
  --wake-hold-max 20 \
  --wake-hold-max-per-key 5 \
  --wake-hold-max-total 40 \
  --wake-keepalive 15s

The placement's hold_budget overrides the default hold budget. Choose it from measured startup time and the application's acceptable wait. Bound the model, key, and total held-request counts together with request-body size, since each held request retains its body in memory. The configuration reference documents these flags and defaults.

Declare the cold model in the placement map

For a new isolated example site, save the following as sibling-map.json. Replace the domain, site details, and model ID with values you operate. The example URL is a placeholder, not a Hivenet endpoint:

{
  "schema": 3,
  "version": 1,
  "region": "eu",
  "sites": [
    {"id": "site-fr-1", "url": "https://site-fr-1.example.com", "country": "FR", "eu": true}
  ],
  "placements": [
    {"model": "example-chat", "kind": "chat", "site": "site-fr-1", "scale_from_zero": true, "hold_budget": "240s"}
  ]
}

A map push replaces the whole map. On an existing deployment, preserve its other sites and placements and use the map writer's version sequence. A lower version is rejected; version 1 here is only for a fresh example. Push the map through the protected admin endpoint, then read it back:

curl --fail-with-body --silent --show-error \
  -X POST "https://site-fr-1.example.com/admin/sibling-map" \
  -H "Authorization: Bearer $ADMIN_KEY" \
  -H "Content-Type: application/json" \
  --data-binary @sibling-map.json

curl --fail-with-body --silent --show-error \
  "https://site-fr-1.example.com/admin/sibling-map" \
  -H "Authorization: Bearer $ADMIN_KEY"

An external map writer must deliver this configuration to every Router instance and restore it after a restart. Router keeps the map in memory. Until it is restored, a request for a model at zero does not enter this hold flow and can fail as model-not-found. Include map restoration in startup readiness and recovery checks.

Connect pending demand to a separate autoscaler

hivenet_pending_demand{model="example-chat"} counts requests still waiting for the model on a Router, including requests released by a wake that have not obtained an agent slot. It stops counting a request once that request obtains a slot; it is not a gauge of all generation in progress. With several routers, sum demand within the same site.

The following adapts Router's documented KEDA example to an existing Kubernetes Deployment. It assumes KEDA is installed, the inference namespace and target Deployment exist, and that Deployment runs the model backend and its agent together. Keep Router, the map writer, and the monitoring/autoscaling components running independently of the model workload.

The Prometheus address must be reachable by KEDA and appropriately protected. This example assumes its selected series cover exactly one site, with each Router scraped once. If your monitoring system covers multiple sites or duplicate scrape paths, add filters using your actual scrape labels before applying it.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: example-chat-wake
  namespace: inference
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: example-chat
  minReplicaCount: 1
  maxReplicaCount: 1
  idleReplicaCount: 0
  pollingInterval: 5
  cooldownPeriod: 1800
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring.svc:9090
        query: >-
          clamp_max(
            (sum(hivenet_pending_demand{model="example-chat"}) or vector(0))
            + (sum(increase(hivenet_routing_requests_routed_total{model="example-chat"}[2m])) or vector(0)),
            1)
        threshold: "1"
        activationThreshold: "0.5"

The example permits one active replica and zero idle replicas. The query combines waiting demand with recent routed requests and caps the result at one. It expresses an on/off signal, not a capacity estimate. The five-second polling interval and 30-minute cooldown are illustrative. Check the KEDA 2.21 ScaledObject specification for the replica and cooldown settings, and its Prometheus scaler reference for query and authentication requirements.

The two-minute activity window adds time before the cooldown begins. Router's reference also notes that increase misses the first sample of a new counter series, so a first request does not establish the same activity history as a warmed series. Measure observed scale-down timing instead of promising an exact idle interval.

The or vector(0) terms handle absent series, but cannot prove that a missing series means no demand. A lost map or failed scrape can remove the signal. Check scrape health and map presence independently, and test the chosen behavior when monitoring fails.

Recent arrivals are also not proof that every earlier request has finished. Before enabling automatic scale-down, verify that the workload's maximum execution time, termination behavior, and any drain controls protect active work. This example does not establish safe termination of arbitrarily long generations or streams.

Make the client recognize failures after keepalives

Keepalives help a held connection survive intermediary idle timeouts. Router sends an SSE comment for streaming requests and whitespace before a non-streaming JSON body. The first keepalive commits HTTP 200 and the headers. A subsequent error must therefore be carried in the body.

OutcomeBefore keepalivesAfter keepalives begin
Model becomes ready and request succeedsNormal success responseHTTP 200, keepalive bytes, then success body
Hold expiresHTTP 503, model_waking, Retry-AfterHTTP 200, then an error envelope or SSE error event; inspect the body retry hint
Hold count limit reachedImmediate HTTP 503 with model_waking and Retry-AfterNo hold begins
Failure after releaseNormal error statusHTTP 200 with the error in the response body

For a non-streaming text chat request, parse the complete JSON body and check its error envelope before treating the response as successful. This small Node.js helper illustrates that boundary:

function readChatResponse(status, body, retryAfterHeader = null) {
  let data;
  try {
    data = JSON.parse(body);
  } catch {
    throw new Error('Response is not complete JSON');
  }
  if (!data || typeof data !== 'object' || Array.isArray(data)) {
    throw new Error('Response must be a JSON object');
  }
  if (Object.hasOwn(data, 'error')) {
    const detail = data.error;
    const code = detail && typeof detail.code === 'string'
      ? detail.code : 'unknown_error';
    const seconds = detail && Number.isFinite(detail.retry_after)
      && detail.retry_after >= 0 ? detail.retry_after : null;
    const headerSeconds = typeof retryAfterHeader === 'string'
      && /^\d+$/.test(retryAfterHeader) ? Number(retryAfterHeader) : null;
    const failure = new Error(`Router request failed: ${code}`);
    failure.code = code;
    failure.retryAfterSeconds = seconds ??
      (Number.isFinite(headerSeconds) ? headerSeconds : null);
    throw failure;
  }
  if (!Number.isInteger(status) || status < 200 || status >= 300) {
    throw new Error(`Unexpected HTTP status: ${status}`);
  }
  if (!Array.isArray(data.choices) || data.choices.length === 0 ||
      typeof data.choices[0]?.message?.content !== 'string') {
    throw new Error('Expected a text chat-completion response');
  }
  return data;
}

module.exports = { readChatResponse };

Pass the HTTP status, complete response text, and optional Retry-After header to readChatResponse. It handles leading whitespace and rejects errors even under HTTP 200. It accepts only text chat completions; tool-call-only results and SSE streams need their own handling. Its retry hint supports numeric seconds, not date-form headers. The helper does not send requests, retry, or validate answer quality.

The 20 offline checks cover success, leading keepalive whitespace, model_waking under HTTP 200 and 503, other error envelopes, retry hints, malformed or truncated bodies, redirects, and responses outside this example's scope. They establish only this helper's behavior with synthetic input, not SDK or deployment compatibility.

For streaming, ignore SSE keepalive comments and recognize the endpoint's error events. Do not interpret the first comment as the first generated token. Router audit records and HTTP metrics can record a final error status even when the wire status was already 200; correlate the body error and request ID using the monitoring guide.

You can set --wake-keepalive 0 to retain normal status codes before a response begins, but only if the full network path tolerates the idle wait. Choose and test the behavior with the actual proxy and client. Keep a total request deadline as well as idle timeouts, since repeated keepalive bytes can prevent an idle timer from firing.

Bound retries and the burst after a wake

A client deadline must allow the accepted cold wait and subsequent work without exceeding the user's overall deadline. If a hold expires, model_waking provides a retry hint. Retry only when another attempt fits the remaining deadline, and stop after cancellation. General retry policy belongs in the fallback guide.

Use a UUID in X-Client-Request-ID and retain it when retrying the same logical request. Router uses the same key and client ID to deduplicate pending-demand accounting on that router. This is not an exactly-once execution guarantee. A retry arriving at another Router instance is counted there too, and an invalid ID is ignored.

When an agent registers, many held requests can be released together. The hold deadline also bounds the released request's wait for global admission capacity. The documented path can return 503 model_waking if the model wakes but no slot becomes available before that deadline; per-key occupancy rejection can still return 429. Verify limits using the existing rate limiting and quotas guide. A longer hold does not add GPU capacity.

Measure cold start latency and validate recovery

First prove a normal request while the model is running. Then exercise one controlled cold wake in a test environment. Record the Router, agent, KEDA, Kubernetes, engine, and model versions alongside configuration, timestamps, request IDs, and observed outcomes. Pin the deployment components you actually test; this guide does not select a GPU or model for you.

ScenarioEvidence to record
Normal cold wakePlacement present, demand rises, autoscaler acts, backend and agent become ready, response is parsed correctly.
Model never becomes readyBounded wait and model_waking handling, both before and after keepalives commit the status.
Burst during startupPer-model, per-key, and total hold bounds; rejected requests; admission after wake.
Cancellation and retryHeld request cleanup, total deadline, UUID reuse, and behavior when a retry reaches another router.
Router restartDynamic-key availability, map restoration, and behavior before the map returns.
Scrape or autoscaler failureDetection of missing or stale signals, operator response, and the resulting replica decisions.
Scale-down with work in progressLong requests and streams finish or fail according to an explicit drain/termination policy; no assumed transparent resumption.

Measure the path from the client’s request to the first generated token, not the first keepalive byte. Record when pending demand appears, when the autoscaler changes replicas, when the model becomes ready, and when the agent accepts work. Cold start latency adds waiting before normal inference; a fast warm response does not establish a fast response after scale-to-zero. Compare cold and warm samples separately with the same model, prompt, and output limit.

For each test, record whether the container image and model weights were already available locally. Repeat with the storage and network conditions your workloads will encounter, including simultaneous wake-ups where relevant. Use those observations to evaluate the hold budget and user deadline. If a traffic surge arrives while the model is cold, test both accepted and rejected requests; reporting only successful responses hides the effect on users who exceeded a hold limit.

Observe pending demand, hold outcomes, hold duration, model readiness, and the client's elapsed time together. The hold-at-zero metrics reference defines hivenet_wake_holds_total and hivenet_wake_hold_seconds. Repeated expired holds call for investigating startup time and the wait budget; full holds call for investigating arrival bursts and configured bounds.

Keep an explicit result for every check: passed, failed, or not run. A correct configuration file cannot establish a measured cold-start time, safe scale-down, or application reliability. Use the results to decide whether this workload should sleep, keep one replica warm, or use another operating arrangement. Hivenet Router supplies the routing and hold behavior described here; the autoscaler and model infrastructure remain part of the deployment you operate.

LLM cold start questions

Does Router reduce model loading time?

Router holds eligible requests and exposes pending demand. It does not load model weights or accelerate the inference engine’s startup. Investigate model loading performance in the backend; configure Router’s hold budget from the startup time you measure.

Should latency-sensitive workloads keep a warm replica?

Keeping a ready replica avoids waiting for that replica to start from zero, at the cost of keeping capacity allocated. Evaluate that trade-off against the application’s first-response deadline and actual billing. A warm model can still queue under load, so test admission capacity as well as cold starts.

Does scale-to-zero guarantee lower inference cost?

No. Zero model replicas can remove idle model capacity, but storage, Router, monitoring, and autoscaling components remain. Compare the complete deployment cost and acceptable latency for the workload. This guide reports no measured cost savings.

How does a cold start affect time to first token?

A request arriving while the model is cold must wait for readiness before generation can begin. That wait increases the client’s time to first token (TTFT). During a traffic surge, users can also encounter hold limits and admission delays after the model wakes. Measure those outcomes separately from warm inference latency and token generation speed.

Does an HTTP 200 mean the cold request succeeded?

No. Router’s first keepalive can commit HTTP 200 before the model is ready. Check the final JSON body or SSE error event and enforce an overall client deadline. A keepalive is not generated model output.

Your next workload belongs on Hivenet.

Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.