Build a RAG pipeline that stays fast at scale

RAG performance depends on relevance and speed together. Slow retrieval adds to response time, while irrelevant evidence can produce an unhelpful answer even when generation is fast. Measure the full path from the user’s request to the completed answer, alongside whether the answer is supported by the retrieved sources.

Chunk boundaries, query planning, reranking, and caching are useful places to test improvements. None is an automatic fix. In vector retrieval, an embedding model converts queries and documents into numerical vectors for similarity search; the model and index settings must suit the material being searched.

Try Compute today

For a self-managed vLLM deployment on Compute, check that the available instance and software environment support your model and serving setup. Follow the current instance setup guidance, then measure TTFT/TPS while you test chunking and rerankers. Do not assume a ready-made vLLM endpoint or a particular location is available.

Introduction to RAG

Retrieval-augmented generation, or RAG, adds information from external sources to a language model’s input. Retrieval can use keyword search, vector search, or both. The model then generates an answer using the retrieved material alongside its learned behavior; retrieval does not by itself make the sources current or the answer correct.

The pipeline has three main stages: prepare and index the source material, retrieve passages the user is authorized to access, and generate an answer from the available evidence. Test each stage. Missing passages, stale documents, and unsupported generation can all cause a poor answer.

Indexing: chunking, embeddings, and vector databases that help, not hurt

Chunk size. Treat 200–400 tokens with 10–20% overlap as an illustrative starting experiment, not an optimum. Check the embedding model’s input limit, preserve meaningful boundaries, and compare retrieval and answer quality on your own questions. Smaller chunks can lose necessary context; larger ones can include unrelated material. See Microsoft’s document-chunking guidance for the tradeoffs.

Boundaries. Split on headings, bullets, and paragraphs to keep ideas intact. Avoid arbitrary character counts.

Normalize. Remove boilerplate only when it is irrelevant to the task. Preserve capitalization, numbers, and whitespace where they affect meaning, especially in code and structured data. Test any normalization before applying it across the corpus.

Metadata. Store source, section, language, timestamp, and access tags for filtering and audits.

Embeddings model. Pick one that handles your languages and domain. Use the distance metric and normalization its documentation specifies, and check truncation limits. Test retrieval on your own labeled query–document pairs rather than relying on leaderboard gaps alone.

Query planning: retrieve less, retrieve better

Retrieve less, retrieve better. Advanced search algorithms, including semantic search, are used to improve retrieval accuracy.

• Hybrid search. Combine BM25 (keyword) and vector result lists using the engine’s supported fusion method. For example, reciprocal rank fusion combines ranks rather than adding raw scores from different scoring systems. Tune the combination on representative queries.
• Filters. Enforce access rules for every request. Choose other metadata filters and their placement using your search engine’s behavior: prefiltering and postfiltering can affect both recall and latency. Measure filter performance instead of assuming filtering first is always faster.
• Candidate count and reranking. As a starting experiment, retrieve k=20–50 candidates and rerank 10–20 of them. These are test settings, not recommended limits for every workload. Measure missed relevant passages, reranker cost, and final answer quality before reducing either count.
• Diversity. De-duplicate near-identical chunks, but retain distinct supporting evidence even when several useful passages come from the same section.
• Multi‑hop queries. If questions span documents, retrieve in two steps: plan → gather → answer.

Reranking that earns its cost

Cross-encoders score query–passage pairs and can improve the ordering of retrieved candidates. Test retrieve-and-rerank against retrieval alone; any relevance gain must justify the added work.

• Batch requests when it improves reranker throughput without breaching your tail-latency target. Measure the effect under realistic load.
• Validate cutoffs. A drop in reranker scores is not automatically a calibrated confidence threshold. Test cutoffs on labeled examples and check what useful evidence they exclude.
• Fallbacks. On reranker timeout, use the already-authorized retrieval results if they meet your quality rules, or return a clear failure or insufficient-evidence response. Log the event without relaxing access controls.
• Measure both token savings and the reranker’s added latency. A shorter prompt can save prefill work, but total time to first token may still increase.

Caching layers that actually help

• Prompt cache. Keep reusable instructions in a stable, identical prefix when the serving system supports it. Prefix caching can reuse prefill work; it does not cache a correct answer or avoid generating new output tokens. Do not change numbers or meaningful formatting just to increase cache hits.
• Retrieval cache. Scope keys to the query, tenant or user access scope, filters, index version, and retrieval configuration. Revalidate authorization and invalidate entries when documents or permissions change; a query hash alone is not an access-control boundary.
• Answer cache. Reuse only answers appropriate for the current user and task, with no unintended personal or private data. Account for source, model, and prompt versions; set an expiry and invalidate on relevant changes. Recheck authorization before serving a cached result.
• KV-cache at inference. Compact context can reduce memory pressure and leave room for more concurrent work. Throughput still depends on the model, hardware, serving settings, and load; verify it rather than assuming a fixed gain.

Latency budgets and SLOs

• Budget split. Set a latency target for your application, then allocate time to retrieval, reranking, queueing, prefill, and generation. Measure client-observed TTFT and full response time at representative concurrency, including p95 and p99. There is no workload-independent 800 ms target or 200–300 ms retrieval allowance.
• Parallelism. Run retrieval and pre‑processing in parallel where safe. Parallel processing is a key technique for reducing latency.
• Async enrichment. Defer nonessential work, such as analytics, when it does not affect the answer. Retrieve and check the evidence needed for a claim before presenting it as supported; do not rely on adding citations after an unsupported answer has already reached the user.

Evaluation metrics: quality and speed together

Build a small, versioned starter set, for example 50–150 representative queries, and expand it as you find gaps. MRR measures how early the first relevant result appears; nDCG considers graded relevance and ranking order. Precision measures the share of retrieved results judged relevant. Answer semantic similarity can compare wording or meaning with a reference, but it is not proof of factual correctness. Track:

• Recall@k and MRR for retrieval.
• Faithfulness: does the answer stick to sources?
• Groundedness: can you cite the exact chunk(s)?
• Latency: TTFT and full response time by route.
• Token use: prompt vs output tokens per request.
• Unsupported and incorrect claims. Label these separately: an answer can agree with a source that is itself wrong. Review source support and factual correctness, and use labeled relevant documents to calculate retrieval recall. Fluency is a readability measure, not evidence that an answer is true.

A/B rerankers and chunk sizes on the same eval. Promote only when both quality and latency improve or hold steady.

Operations: runbooks and observability

• Metrics. Request rate, TTFT, TPS, retrieval latency, reranker latency, prompt tokens, output tokens.
• Logs. IDs, counts, and source references; avoid raw text by default.
• Incidents. Drill on vector index rebuilds, reranker outages, and cache stampedes. Significant challenges can arise during data retrieval and processing the original query, especially during outages or large-scale updates.
• Data changes. On bulk updates, re‑embed in batches; keep two indices for blue/green swaps. Automated frameworks like RAGAS and TruLens provide automated metrics for assessing retrieval and generation quality in RAG systems.

Try Compute today

For a self-managed vLLM deployment, choose from the Compute locations and resources currently available in the console, and verify software compatibility. Stream tokens and set output caps appropriate to the task, then measure the full pipeline. Location choice and retrieval tuning do not guarantee faster first tokens.

Benefits and Challenges

RAG can help answer questions using a corpus that is too large or changes too often to include in every prompt. Its benefit depends on retrieving useful, authorized evidence and using it correctly. Retrieval and reranking also add work, so compare quality, latency, and cost against a simpler prompt-based approach before expanding the system.

Scaling requires maintained sources, tested retrieval, and clear access rules. Human review can catch answer-quality issues that automated checks miss. In a December 2024 storage-demand forecast, McKinsey cited an uncertain expert expectation that about 25% of large enterprises would adopt RAG by 2030. That is a forecast assumption, not measured adoption or evidence that a particular pipeline will meet its targets.

Keep retrieval augmented generation fast with smart retrieval and short prompts

Test chunk boundaries, search methods, and rerankers against the same questions. Cache reusable work only where source freshness and access rules allow it. Measure retrieval quality, TTFT, full response time, and token counts together before adopting a configuration. For complex questions, test query rewriting or additional retrieval steps and include their cost in the comparison.

Last thoughts

RAG adds external evidence to a language model’s input, but it cannot guarantee a current, correct, or trustworthy answer. The index can miss a useful passage, the source can be outdated, and the model can make an unsupported claim. Keep those failure modes visible in your tests and monitoring.

Maintain a versioned evaluation set, update sources and permissions, and retest when models or retrieval settings change. Keep a configuration because it meets the application’s quality and performance requirements, not because RAG is assumed to be cheaper or more reliable.

FAQ

What chunk size works best for RAG?

Treat 200–400 tokens with 10–20% overlap as an illustrative trial, not a universal recommendation. Compare meaningful chunk boundaries and different sizes on your own queries, respecting the embedding model’s limits. Smaller chunks are not always better for recall or answer quality.

Should I always use a reranker?

Use one when its measured relevance gain justifies the extra latency and cost. Time it with your model, candidate count, hardware, and concurrency; there is no general 10–30 ms allowance. For simple FAQs with good retrieval, a reranker may add little value.

How many chunks should I pass to the LLM?

You can test 5–10 as a starting count, but select the final limit using evidence coverage and the prompt’s token budget. A count alone does not describe how much text the model receives. Check that limiting chunks does not remove support needed for the answer.

How do I handle multilingual corpora?

Use compatible multilingual query and document embeddings, or maintain language-specific indexes when appropriate. Test cross-language retrieval on real questions; do not assume a separate vector represents an entire language. Prefer sources in the user’s language when they provide the needed evidence, without excluding a better source solely because of language.

Is long context simpler than RAG?

For a small, controlled document set that fits within the model’s limits, long context can avoid an indexing and retrieval pipeline. It is not always slower or more expensive. Compare both approaches on the same questions, quality requirements, traffic, and full operating costs.

For a workload-level comparison, see our long context vs. RAG guide. Compare both approaches with the same questions, access rules, and quality requirements before choosing.

How do I prevent outdated answers?

Synchronize source edits, deletions, and permission changes with the index and caches. Show source dates when useful, and use recency filters only when the question calls for current material. A recent timestamp alone does not make an answer correct; check that the retrieved source is authoritative for the question.

‍