
RAG reranking adds a scoring step between retrieval and answer generation in a retrieval-augmented generation (RAG) pipeline: a model compares a query with the candidate passages and returns a new order. With Hivenet Router, an application sends those candidates to POST /v1/rerank; Router selects an eligible reranking agent, and the backend model scores them. The application keeps responsibility for document permissions, source identifiers, and the passages it eventually sends to an answer-generating model.
This guide is for developers who already have a retrieval step and want to connect a self-hosted reranker. It follows one small request from an authorized candidate list through Router and back to the original documents. For chunking, retrieval strategy, and overall latency budgets, use the existing RAG performance guide.
Validation scope: Configuration was checked on November 18, 2026 against Hivenet Router v0.1.7, the published Infinity Python package 0.0.80, and the linked official documentation. The response-mapping example passed 18 offline checks on Node.js 24.22.3 using synthetic responses. No Router-to-Infinity deployment or model-quality benchmark was run. The commands below are a documented setup to validate in your environment, not an executed deployment result.
In bi-encoder vector search, the model creates separate embeddings for the user query and the documents. The retrieval system compares those vectors to select candidates. A cross-encoder reranker processes the query and each candidate together, producing a relevance score for that pair. This extra computation makes the candidate count an operational choice: use the reranker on the retrieved set rather than scoring the entire knowledge base for every request. See the Sentence Transformers cross-encoder documentation for the model behavior.
For this Router integration, keep the RAG pipeline responsibilities explicit:
A reranker can reorder the passages your retriever found. It cannot recover a useful document that never reached the candidate list, repair missing permissions, or establish that the final answer is correct. Start with questions for which useful passages are already present but poorly ordered.
For example, a password-reset query might retrieve a recovery runbook, an account-help page, and a billing page. The next step is to score those candidates against the query while retaining their document IDs and source paths. The Sentence Transformers retrieve-and-rerank guide explains this two-stage pattern. Whether it improves your application needs a comparison on your own questions.
Keep the first version narrow: one reranking model, one backend, one dedicated client key, and a small candidate set. Router becomes useful when the application needs its routing and access controls or several eligible backends. A single application can also call a backend directly; adding Router does not itself improve the relevance scores.
The Router v0.1.7 Infinity integration documents the Python v2 server. The Infinity README reviewed on November 18, 2026 introduces a separate Rust-based v3 server, whose quickstart and reranking path differ. This example deliberately uses infinity-emb 0.0.80 and the infinity_emb v2 command. Do not substitute a moving image tag or v3 command and assume the same integration contract.
The example runs Infinity and the agent on the same Linux host. Router is already running on a reachable host, with client authentication configured. You need Python 3.11–3.13 for this Infinity package; the commands use Python 3.12, a virtual environment, and an NVIDIA CUDA setup suitable for the model. They do not establish a minimum GPU size or a throughput target. Installation also needs access to the package and model sources.
Install the pinned Python server in an isolated environment, then start one reranking model. The server runs in the foreground; leave this terminal open and use another shell on the same host for the health checks and agent command:
The model revision is pinned for repeatability. BAAI/bge-reranker-large is an example from Router's integration documentation, not a recommendation for every language or corpus. Review its model card, input limits, license, and results on your material before choosing it.
The /v1 prefix places model discovery at /v1/models and reranking at /v1/rerank. In the checked Python package, health remains at /health. Keep Infinity bound to loopback for this same-host layout; an agent in another container or host has a different network path. Do not expose the unauthenticated backend publicly to make a connectivity check pass.
Confirm that the model list contains bge-reranker-large. The source repository ID tells Infinity which weights to load; the served name is what the agent and client must request. The optional --batch-size flag is omitted so this guide does not present a workload-independent tuning value.
Use a Hivenet agent from the same v0.1.7 release as Router. The following command assumes its binary is installed, the shared agent secret already exists in a protected file, and the identity directory is writable by the service account. Replace the reserved example hostname with your Router's address. The agent must reach Router's authentication and transport listeners; the Router quickstart covers the initial deployment.
Three settings must agree: Infinity's served model name, the agent's --model, and the request's model. The capability value is reranker, while the HTTP path ends in rerank. The agent's backend URL is the origin without /v1; the integration supplies the endpoint paths.
The agent also runs in the foreground. Keep it running while you make client requests from another shell. Keep the identity file across restarts and give each agent its own path. An embedding model in the same Infinity process needs a separate embedding agent. If your Router policy filters on region or tags, supply metadata that satisfies that policy. Do not add the deprecated --capacity flag as a concurrency control: Router v0.1.7 ignores it for network-connected agents.
Use a client key permitted to call this exact reranking model. That key is different from the shared agent secret and an administrator key. Confirm the model and its reranker capability through Router's GET /v1/models using the same client key that will make the request. If it is absent, check registration, backend health, and model restrictions before changing the client payload.
Filter documents by the caller's permissions before sending their text to the reranker. Keep a stable array for the life of each request, with the text and source metadata together. This three-document fixture is synthetic:
Save it as candidates.json. In an application, those records would come from the existing retrieval step. Do not silently sort or filter the array after sending it: the returned indexes refer to the order you sent.
The following Bash commands use jq and curl. Set ROUTER_ORIGIN to your approved HTTPS origin, with no trailing slash, and supply ROUTER_API_KEY through your approved secrets process. The example origin is a placeholder. The 20-second client deadline is an example to adjust within your application's overall deadline.
The version-pinned reranking reference defines model, query, documents, and top_n. Router validates the model field and routes the request; the backend validates the query, document list, result count, and optional fields. A client designed only for Chat Completions does not establish reranking compatibility.
top_n: 2 requests at most two returned results. It does not mean that only two candidates are scored. Bound the input list and passage sizes separately. This request uses the ordinary synchronous response, not server-sent events, and it does not assume that the backend returns document text.
A result's index identifies its position in the submitted document array. Its document field may be null. Keep your original IDs, sources, and text instead of accepting response-supplied text as a replacement. The following response is a hand-written fixture for checking that mapping, not a measured model output:
Save the fixture as rerank-fixture.json. Then save this script as rerank-map.mjs. It requires Node.js, uses only built-in modules, and does not make network requests:
Run node rerank-map.mjs candidates.json rerank-fixture.json 2. The output should preserve account-02 and its source first, followed by recovery-01. Once a real request succeeds, replace the fixture filename with rerank-response.json.
The script rejects missing result lists, excess results, duplicate or out-of-range indexes, non-integer indexes, and non-finite or non-numeric scores. It preserves the backend's result order and does not assume that every score is a probability between zero and one. The 18 offline checks also confirmed that candidate records stay unchanged and response-supplied document text cannot replace them. An empty result list is returned as empty; the calling application must decide whether to stop or use a separately approved fallback.
This validates the response contract, not relevance. The function also does not perform authorization: its input must already be permitted for the caller. Recheck access when material is served if permissions may have changed during the request. Carry the source IDs into the answer-generation step so citations can refer to the passages actually selected.
Reranking cost depends on the number and length of query-document pairs, model, hardware, and competing work. In Router v0.1.7, reranking participates in request-rate controls but does not charge query or document tokens against the daily token quota. It is also exempt from the generation-specific occupancy gates. A chat token budget is therefore insufficient protection for this endpoint. The Infinity integration relies on the backend's batching and scheduler alongside request-rate and body-size controls.
Set application limits on candidate count, passage length, total request size, and concurrent reranking work. Router's documented default request-body limit is 10 MiB, including the complete JSON payload; your proxy or model may impose a smaller limit. A client timeout does not prove that all downstream computation has stopped. Check cancellation and backend load in the deployment before using aggressive retries.
Choose the fallback behavior before an incident. An application may use its original, already-authorized retrieval order when that meets its quality requirements, or return a clear failure or insufficient-evidence result. Do not silently broaden document access, remove tenant filters, or send the passages to another provider. Router v0.1.7's OpenAI/Anthropic provider fallback does not handle reranking. For compatible local alternatives, validate the same model and reranker capability against the fallback guide.
Use the quotas guide for the wider control model and the Router security guide for endpoint access and permitted destinations. Keep document bodies and credentials out of routine troubleshooting logs.
First check the integration with harmless inputs: model discovery, a direct backend request, the same request through Router, correct source mapping, and clear handling of invalid credentials, missing models, oversized requests, and timeouts. Test a stopped backend separately from a stopped agent. Record what actually happened; an unchecked case remains an acceptance test to run.
Then compare retrieval alone with retrieval plus reranking using the same questions, candidate lists, corpus snapshot, and access rules. Check whether known relevant passages remain in the final selection, whether citations still map to the right documents, and what happens when none of the candidates is useful. A high relevance score is not evidence that the passage is true or sufficient to answer the question.
Measure reranking duration and the complete user request under representative concurrency. Reducing the final context can reduce downstream work while the additional reranking step increases total latency. Keep the comparison in the existing RAG performance workflow; do not infer a quality gain or latency saving from this three-document fixture.
For operational evidence, capture the request ID, model, candidate count, status, and duration without retaining the candidate text by default. Router v0.1.7's Infinity integration does not scrape engine-specific Infinity metrics. Use application timing, backend evidence, and the Router monitoring guide to investigate missing or slow results.
No. It scores a candidate set after retrieval. The first stage may use keyword search, vectors, or both. Keep retrieval improvements separate in the evaluation so you know which change helped.
Choose the input candidate count and the final result count separately. The three candidates and top_n: 2 here demonstrate the API, not a production recommendation. Compare candidate counts using the same labeled questions, then check whether the final selection retains the evidence needed to answer them within your latency and context-window budgets. A larger input list adds query-document pairs to score; a smaller top_n only limits the returned results.
No. Reranking changes which retrieved passages are selected for the language model; it does not verify the sources or guarantee that the generated answer follows them. Test answer support separately, preserve source references, and handle questions for which the retrieved documents contain insufficient evidence.
The reranking backend generates them. Router forwards the successful response without changing its scores or order. It does not search your document store or decide which passages a user may access.
Router v0.1.7 rejects the combination of the Ollama engine and reranker capability. This guide uses the documented Infinity integration. Another backend must satisfy the actual reranking endpoint contract; an OpenAI-compatible chat endpoint alone is not sufficient.
That has not been established by this guide. The reviewed Router configuration uses Python v2 with the explicit /v1 prefix. Compare health, discovery, reranking paths, model naming, and responses before changing server generations.
Start with this bounded request path, prove the document mapping, and evaluate the ranking change before expanding the workload. The routing guide explains how the same Router deployment can sit in front of other supported inference capabilities.
Pick one AI, compute, or storage workload and see the difference for yourself. Spin it up in minutes, or let our team map your fastest path to production.