LLM monitoring: troubleshoot Hivenet Router requests

Useful LLM monitoring starts with a request someone could not complete. Record its time, model, response status, and request ID, then follow the evidence through the router, the selected agent, and the inference backend. A green health endpoint or a low average latency cannot explain an individual failure.

Hivenet Router routes requests to inference engines you operate. This guide is an investigation sequence for an existing deployment, covering unavailable agents, full capacity, quota rejections, and slow responses. It does not evaluate the quality or factual accuracy of generated answers.

Validation scope: The documentation and relevant source were checked on November 11, 2026, against Router commit f97fec83. The offline audit-file example passed 26 checks on Windows CPython 3.12.14 using synthetic records. No live Router, inference backend, Prometheus, Loki, or trace collector was run for this guide. The operational steps and PromQL expressions require validation in your deployment.

Start with one failed request

Keep the original response status, structured error body, and response headers before retrying. Also record the client-observed duration, whether streaming was enabled, the model or alias requested, and the time window with its UTC offset. Avoid putting prompts, authorization headers, or credentials into a shared incident note.

In the checked version, Router generates the ID returned in the X-Request-ID response header. A valid client-supplied UUID is retained separately as client_request_id. Read the response header rather than assuming the ID you sent is the Router's ID. Repeated client IDs can connect attempts, but each attempt has its own Router request ID. The audit logging reference describes these fields.

Use a matching trace ID only if tracing was active and the spans were collected. An empty trace field does not mean the request never ran. Likewise, a missing selected agent can be consistent with rejection before dispatch.

Separate process health from model readiness

Check the following surfaces in order. These are read-only observations; do not reset counters, alter quotas, or restart agents before preserving the incident evidence.

• GET /health confirms that Router's HTTP process responds. It can succeed while no backend can serve the requested model.
• GET /admin/health summarizes connected agents, their health, and the global request channel. It is not proof that the requested model has free capacity.
• GET /admin/routing-table exposes the current agent state. Inspect the requested model, metadata, backend health, active requests, and declared capacity, then consider the request's routing policy.

Use an authorized administrator connection for the admin endpoints and follow the admin API reference. A healthy agent can still be ineligible because it serves another model or fails a policy condition. A snapshot taken after recovery also cannot prove what was available during the failure; compare it with historical logs and metrics.

Use the domain error to choose the next check

HTTP status alone is too broad. The Router error body's code and source narrow the investigation. The error reference documents the complete mapping.

• unauthorized or model_forbidden: check the presented credential and model access. Adding inference capacity will not correct an authorization problem.
• rate_limit_exceeded: inspect the key's quota configuration and applicable admission limit. A missing strict per-model quota declaration can also produce this code; it does not always mean a burst of traffic.
• no_agents_available: examine model eligibility, agent connection state, backend health, and policy exclusions.
• no_capacity: eligible agents exist but their declared slots are full. Check concurrency, the capacity queue, and how long requests occupy slots.
• request_timeout: inspect the request timeline to locate the delay. The deadline expiring does not by itself identify the slow component.

Preserve an unfamiliar code rather than forcing it into one of these categories. Backend errors, admission controls, and model-wake behavior have additional codes. Check source before attributing a response to Router itself.

Find the matching audit record without exporting the whole log

For a small local investigation, the following standard-library script selects one exact Router request ID and prints only an explicit set of fields. Save it as find_request.py. It reads a local UTF-8 JSONL file without modifying it. It does not fetch logs or contact Router.

"""Read a local audit JSONL file; print selected fields for one request ID."""
import argparse
import json
import sys

FIELDS = ("request_id", "client_request_id", "model", "status_code",
          "error_code", "latency_ms", "agent_id", "trace_id")


def find_records(lines, request_id):
    matches = []
    malformed = 0
    for line in lines:
        if not line.strip():
            continue
        try:
            record = json.loads(line)
        except json.JSONDecodeError:
            malformed += 1
            continue
        if not isinstance(record, dict):
            malformed += 1
            continue
        if (record.get("log_type") == "audit"
                and record.get("request_id") == request_id):
            matches.append({key: record.get(key) for key in FIELDS})
    return matches, malformed


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("path")
    parser.add_argument("request_id")
    args = parser.parse_args()
    with open(args.path, encoding="utf-8") as stream:
        records, malformed = find_records(stream, args.request_id)
    print(json.dumps(records, indent=2, ensure_ascii=False))
    if malformed:
        print(f"Skipped {malformed} malformed records", file=sys.stderr)
    return 2 if malformed else (0 if records else 1)


if __name__ == "__main__":
    raise SystemExit(main())

For a reproducible exercise, save these two synthetic records as example-audit.jsonl. They illustrate a failed attempt followed by a successful attempt sharing a client ID. The durations, identifiers, and model are invented; this is not captured Router traffic or a benchmark.

{"log_type":"audit","request_id":"019a7250-0000-7000-8000-000000000001","client_request_id":"11111111-1111-4111-8111-111111111111","model":"demo-model","status_code":503,"error_code":"no_capacity","latency_ms":1200,"agent_id":"","trace_id":""}
{"log_type":"audit","request_id":"019a7250-0000-7000-8000-000000000002","client_request_id":"11111111-1111-4111-8111-111111111111","model":"demo-model","status_code":200,"error_code":"","latency_ms":800,"agent_id":"demo-agent","trace_id":""}

Run the lookup for the first attempt:

python find_request.py example-audit.jsonl 019a7250-0000-7000-8000-000000000001

This is the captured output from the offline script:

[
  {
    "request_id": "019a7250-0000-7000-8000-000000000001",
    "client_request_id": "11111111-1111-4111-8111-111111111111",
    "model": "demo-model",
    "status_code": 503,
    "error_code": "no_capacity",
    "latency_ms": 1200,
    "agent_id": "",
    "trace_id": ""
  }
]

The example isolates the 503 attempt without merging it with the later 200 response. An empty agent field does not name a faulty backend. The no_capacity code points toward capacity investigation, but the record alone cannot establish why slots stayed occupied.

Exit code 0 means a match was found in a clean input file, 1 means no match, and 2 means malformed records were skipped. Missing fields remain null and duplicate records remain visible. Use a bounded incident excerpt for large logs; the script scans the file and retains matching rows. For centralized history, use Loki or your existing log search with the same request ID. The field selection limits accidental output, but does not replace access controls or redact sensitive text inside a selected field.

Check whether the monitoring data is available

Before interpreting a graph, confirm the scrape target is up. In the examples below, job="hivenet-router" refers to the Prometheus scrape-job label; replace it if your configuration uses another name. Evaluate each expression separately.

up{job="hivenet-router"}

A zero value means that Prometheus could not scrape that target. An absent series may mean that the target or labels differ. Neither case is a measured zero-error workload. Router exposes /metrics on a separate port, 2112 by default, without built-in authentication. Keep it reachable only through the intended private network or access controls. See the Router metrics reference.

Then inspect requests Router could not forward, broken down by Router instance and model:

sum by (instance, model) (
  rate(hivenet_routing_requests_failed_total{job="hivenet-router"}[5m])
)

This is a per-second routing-failure signal, not the application's total error percentage. Authentication failures, backend responses, and interrupted streams need their own evidence. The Prometheus function reference explains why counters should be evaluated with rate over a time window. Apply rate before summing so resets are handled per series.

Investigate slow responses at the right layer

For requests waiting on declared agent capacity, inspect the per-model Router queue:

hivenet_queue_depth{job="hivenet-router"} > 0

Compare that with hivenet_agent_capacity_utilization and the backend's hivenet_agent_engine_waiting_requests. Router capacity waiting and an inference engine's internal queue are different stages. Increasing a declared slot limit without checking backend behavior can move the wait rather than remove it.

Engine metrics vary by backend and may be absent until reported. A stale hardware or engine snapshot can also outlive a collection problem. Check heartbeat freshness and collection logs before interpreting an unchanged value as stable performance. The engine metrics guide explains backend coverage.

Use client duration for the user's wait, audit latency for Router's completed handling time, and available traces to separate stages. SRTT is a smoothed Router-to-agent/backend signal, not client round-trip time or first-token latency. For metric definitions and benchmark design, use our separate LLM inference metrics guide.

For streaming, retain evidence that the response completed. An HTTP 200 can precede an incomplete stream. Inspect the terminal event and relevant application errors; the checked Router version also exposes hivenet_stream_truncated_total. A low HTTP error count does not establish that users received complete answers.

Close the incident with evidence

After correcting the identified cause, repeat the same request shape in a controlled test. Record the new request ID, outcome, relevant queue state, and end-to-end duration. If the proposed fix is more capacity, confirm that waiting and errors improve without creating memory pressure or a slower backend queue.

Keep a short incident record: affected time window, models, observed error codes, supporting logs, the change made, and the recovery check. Set alert thresholds from the service's own baseline and user requirements. Protect audit metadata even though the dedicated audit record excludes prompt and response bodies; other components can log different data. Hivenet Router provides operational evidence, while the investigation still depends on correlating the right signals.