LLM inference in production: a practical guide

Training gets the attention. Inference carries the load. Traffic is spiky, prompts vary in length, and people expect words on screen almost immediately. To keep that promise, you need a serving setup that treats memory, batching, and cost as first‑class concerns. There is always a balance between minimizing latency and maximizing throughput when optimizing LLM inference. Low latency is critical for interactive applications to ensure a good user experience.

‍
Need control over the serving stack? Compute provides GPU or CPU instances where you configure and operate your model server. For a managed model endpoint, choose Hivenet Inference API and check its supported models, variants, and regional capacity.

Why inference is hard

Requests arrive in bursts and prompts vary in length. For an autoregressive Transformer, the KV cache is built during prefill and grows during decoding. On a GPU deployment, weights, cache, and engine workspace share VRAM. Measure how memory limits, computation, and queueing affect your workload.

Your goal is simple to say: keep latency low while serving as many tokens per second as your users need, without blowing the budget. Balancing latency and throughput is critical when optimizing LLM inference, as both metrics significantly impact performance and cost. Evaluating LLM performance involves monitoring these metrics to ensure efficient and cost-effective operation. One of the biggest challenges of LLM inference is its computational cost, which can lead to high latency and expense. Latency is crucial for user experience in interactive, real-time applications.

Metrics to watch

These are key metrics typically measured when evaluating LLM inference performance:

• Time to first token (TTFT). How quickly the response starts. Users feel this, as it marks the initial delay before the model's output begins. TTFT is influenced by request queuing, prefill, and network latency.
• Tokens per second (TPS). State whether you mean the output rate of one response or aggregate server throughput.
• Throughput. Measure completed requests or output tokens per unit of time under defined latency targets. Concurrent requests describe concurrency, not throughput.
• Intertoken latency (ITL). The average time between the generation of consecutive tokens in a sequence, which impacts the smoothness of text generation. The time to generate each completion token directly affects overall inference speed and the responsiveness of the model's output. Acceptable latency varies depending on the use case; for example, chatbots require lower latency than offline processes.

Track end-to-end latency and its percentiles too. Align metric definitions and use the same model, tokenizer, and request profile when comparing results. Different tokenizers make raw TPS comparisons unreliable.

Choose compute for your workload

Compare available GPU and CPU presets, choose a compatible image, and review the active price before launching. Configure and test your application on the selected instance.

Try Compute now

Serving architectures

Single GPU. Straightforward for 7B–13B models, proofs of concept, and small apps.

Multi GPU. One host, several cards. Use tensor or pipeline parallelism to fit larger models or raise throughput. As concurrent requests increases, higher throughput can be achieved up to the limits of the inference system.

Horizontal scale. Many nodes behind a gateway. Add load balancing, sticky sessions for cache reuse, and a scheduler that knows about prompt and output lengths. Load balancing and scheduling are essential for scaling LLM systems efficiently.

Serverless endpoints. Good for sharp spikes when you can accept cold starts and variable cost.

Current self-serve Compute GPU presets use RTX 5090 with 32 GB per GPU; the RTX 4090 fleet is retired. Check the current GPU options and active prices in the console, and verify that your chosen runtime supports the selected hardware.

Engines at a glance

vLLM. Strong concurrency from continuous batching and smart KV‑cache paging. Ships an OpenAI‑compatible HTTP server.

Text Generation Inference (TGI). Hugging Face now lists TGI as being in maintenance mode. Account for its maintenance status when choosing an engine for a new deployment.

TensorRT-LLM. An NVIDIA inference toolkit with hardware-specific optimizations. Verify support for your model and configuration, then benchmark it rather than assuming it will be the fastest option.

Ollama. Great locally or for simple single‑box setups. Less focused on high‑traffic APIs.

Pick based on traffic profile, model support, and how much tuning you want to own.

Context and memory

Longer inputs and outputs can increase KV-cache memory and latency. Requirements depend on model architecture, precision, and active requests. Size weights, cache, and workspace together, and test the longest contexts and busiest traffic you plan to support.

• Retrieval-augmented generation (RAG). Retrieve relevant passages rather than sending an entire collection. Retrieval does not automatically shorten a prompt: cap the retrieved context and evaluate whether it supports the answer.
• Efficient caching. Paged KV storage manages attention state in blocks. Prefix caching can reuse matching prompt prefixes and reduce prefill work; it does not reuse arbitrary similar passages or remove the cost of generating new output.
• Quantization. Lower-precision weights, and where supported activations or cache, can reduce memory requirements. The format determines what is quantized. Check engine support and test representative requests.

Quantization, briefly

Lower precision can save memory, but throughput and quality changes depend on the method, kernels, model, and workload. Check supported quantization formats and hardware. Compare against a higher-precision baseline on your own tasks; do not assume quality loss will be small.

Hardware choices

• Memory first. VRAM sets model size and batch depth. Twenty‑four gigabytes fits many 7B models with room to batch. Larger models or long contexts often need 48–80 GB or multiple GPUs, which increases infrastructure costs as more hardware is required.
• Compute next. Extra cores help during prefill. Batch well and decode stays efficient.
• Network placement. Put the endpoint near users. Network latency stacks quickly. FlashAttention reorders the attention mechanism’s computations to reduce memory bandwidth requirements.

Optimizing hardware selection and batching strategies is essential for maximizing cost efficiency in LLM inference, balancing performance with resource and infrastructure costs.

Check available locations for the service and model you intend to use. Compute instance locations and Inference API model availability can differ. For Inference API, inspect the endpoint details to confirm placement; selecting Global lets Hivenet assign an available location.

Costs that matter

• GPU hours. Your main line item. Right‑size hardware to model and traffic.
• Idle time. For a self-managed deployment, use only the scaling and shutdown controls your stack supports. Hivenet Inference API currently uses fixed replicas without autoscaling. Stop an endpoint when it should be offline, and review the billing effect and capacity available before restarting.
• Token waste. Long prompts and high max_tokens burn money. Stream responses and cap outputs.
• Batch size. Higher batch sizes can lead to more efficient GPU utilization but may increase latency for individual requests. Larger batch sizes can improve throughput but generally lead to increased latency. Streaming mode allows LLMs to provide incremental outputs, enhancing user experience. Maximizing throughput with optimal batch sizes can significantly improve cost efficiency by making better use of available hardware resources.

Balancing latency and throughput is critical when optimizing LLM inference, as both metrics significantly impact performance and cost efficiency.

A rough model: estimate daily tokens generated, divide by expected tokens per second per GPU, then convert to GPU hours. Compare with real traffic and add headroom for spikes. As concurrency increases, total tokens per second (TPS) grows until a saturation point is reached, beyond which performance can decrease. It's important to understand how many requests your system can handle within a given time frame to plan for capacity and manage costs effectively. Note that real world performance may differ from these estimates due to hardware variations and infrastructure factors, so always validate with actual deployment data.

Reliability and observability

LLM performance benchmarking and evaluating LLM performance using key metrics are critical for ensuring reliable and efficient deployments. Tracking these metrics helps teams understand system capacity, identify bottlenecks, and optimize resource usage.

Track at least:

• Request rate, queue length, TTFT, TPS
• GPU memory allocation, KV-cache occupancy and preemptions; prefix-cache reuse when enabled
• Model inference resource usage (GPU utilization and memory requirements during inference)
• Error types: OOM, timeouts, 5xx
• Monitoring Model Bandwidth Utilization (MBU) can help compare efficiency across different inference systems.

Common benchmarking metrics include time to first token (TTFT) and tokens per second (TPS), which are essential for evaluating system performance. Benchmarking LLMs is essential for assessing their performance and efficiency in real-world applications, helping teams identify areas for improvement and optimization. Evaluating the performance of LLMs involves using various tools that define, measure, and calculate metrics differently. Performance benchmarking helps identify issues related to model efficiency and optimization. Combining load testing and performance benchmarking provides a comprehensive understanding of LLM deployment capabilities. Analyzing the latency curve is also important to understand the trade-off between batch size and latency, and how different configurations affect throughput and response times.

Alert when TTFT rises or TPS falls under steady load. That is often a signal of memory pressure, bad batching, or performance bottlenecks.

Security and data residency

Use encrypted connections, rotate keys, restrict access, and define what the application logs and retains. Check the locations used by the endpoint and any retrieval, storage, monitoring, and backup services; an EU endpoint location alone does not establish that the whole application keeps data in the EU.

Configure access before serving traffic.
For a Compute instance, enable the required connection option and verify the server, port, and firewall settings. For Hivenet Inference API, use the endpoint URL, model value, and API key shown for that endpoint. HTTPS does not replace authentication or access controls.

Build or buy

Choose Compute when you want to configure and operate the instance, runtime, and model server yourself. Choose Hivenet Inference API when a supported managed model and variant fit your application. Follow the endpoint request instructions, including the exact URL, model value, and organization API key. Test the API routes, request options, and response behavior your application relies on before switching traffic.

Further Reading

• ‍Switch from OpenAI without breaking code‍
• Choose a serving engine‍
• Continuous batching
• Long context vs RAG
• Quantization without wrecking quality
• The right GPU for your workload
• Metrics that matter
• EU privacy checklist
• Production checklist
• Add streaming the simple way
• RAG at scale
• Rate limiting & quotas
• Multi-GPU serving
• Fix TTFT/TPS
• Tokenization traps
• Buyer’s guide to OpenAI-compatible APIs

FAQ

What is TTFT and why does it matter?

Time to first token is the gap between sending a prompt and seeing the first token. Short TTFT improves perceived speed and trust. People feel this number more than any other. End-to-end request latency (e2e_latency) includes the time from when a request is sent to when the final token is received, providing a broader measure of user experience.

How many concurrent users can one GPU serve?

It depends on model size, context length, and batching. A well‑tuned 7B model with short prompts and streaming can serve many users on a single 24 GB card. Long contexts cut that number quickly.

Does a long context beat RAG?

It depends on the application. RAG retrieves selected evidence, while a long-context request supplies more material directly. Both consume context tokens. Compare answer quality, retrieval failures, latency, and cost; a hybrid may be useful.

Do I need multi‑GPU right away?

Start single GPU if you can. Move to multi‑GPU when memory or throughput demands it. Test parallel modes and watch cache health.

Can I keep data in the EU?

If EU-only processing is a requirement, select an explicitly available EU location and verify the endpoint details. Check the locations and retention settings of the rest of your application as well. In Hivenet Inference API, Global lets Hivenet assign an available location, so do not use it as an EU-only placement guarantee.

What is a LLM inference?

LLM inference is the process where a large language model generates a response based on an input prompt by processing tokens through its neural network. During inference, the LLM processes the prompt by activating its vast network of parameters to predict the most likely sequence of tokens. LLMs can process large volumes of text and provide concise summaries of articles or documents.

What are the stages of LLM inference?

LLM inference typically involves two stages: the prefill phase, where the input tokens are processed, and the decoding phase, where the model generates output tokens one at a time.

What is the difference between inference and training LLM?

Training involves adjusting the model's parameters using large datasets, while inference uses the trained model to generate outputs without changing its parameters. LLMs can generate articles, stories, marketing copy, and even code.

What are LLM inference engines?

These are software systems designed to efficiently run LLMs for generating outputs, optimizing for latency, throughput, and resource usage.

What is vLLM for?

vLLM is an inference engine focused on strong concurrency with continuous batching and efficient key-value cache management to optimize LLM serving.

What is the difference between vLLM and LLM?

LLM refers to the large language model itself, while vLLM is an engine or framework for serving LLMs efficiently in production.

Is vLLM faster than Ollama?

There is no universal winner. Compare the same model, precision, hardware, request lengths, and concurrency. Engine features and defaults can affect memory, latency, and throughput.

Why is vLLM so fast?

Because it uses continuous batching and smart key-value cache paging to maximize GPU utilization and reduce latency.

What does LLM serving mean?

LLM serving refers to deploying and running large language models to respond to user requests in real time or batch modes.

What is an LLM serving engine?

It is a platform or software that hosts and manages LLMs, handling inference requests efficiently.

What is an LLM server?

A server configured to run LLM inference workloads, providing access to model predictions via APIs or other interfaces.

What does LLM as a judge mean?

It refers to using LLMs to evaluate or score outputs, such as assessing model quality or ranking responses.

What is tokens per second?

Tokens per second (TPS) measures how many tokens an LLM generates or processes in one second, indicating throughput.

How many words is 1,000 tokens?

There is no fixed word count. It depends on the tokenizer, language, punctuation, and content. Count tokens with the model's tokenizer when planning context or cost.

What does a token mean in AI?

A token is the smallest unit of text that a language model processes, which can be a word, subword, or character.

What is TTFT?

Time to first token (TTFT) is the latency from sending a request to receiving the first generated token.

How to measure TTFT?

By recording the time difference between submitting a prompt and receiving the first output token from the model.

What is the TPOT metric in LLM?

Time per output token (TPOT) measures the average time taken to generate each output token after the first one.

What is a KV cache?

A Transformer's KV cache holds attention key and value tensors computed during prefill and decoding so later tokens can reuse that state.

What is GPU KV cache?

It is the storage of key-value cache data within GPU memory to accelerate LLM token generation.

What is the KV cache in LLM?

The KV cache holds keys and values from previous tokens to efficiently compute attention for new tokens.

What is key-value store cache?

A key-value store maps keys to values for lookup. A Transformer's KV cache stores attention tensors. The shared abbreviation does not make them the same mechanism.

What is continuous batching?

A scheduler updates the active requests between generation iterations as capacity becomes available. This can improve utilization, but queues and memory limits still apply.

What is a continuous batch?

The active batch changes between iterations: completed requests release resources and waiting requests may be admitted. It is not a guarantee of immediate admission or unlimited capacity.

What is the difference between continuous batching and in-flight batching?

TensorRT-LLM documentation also calls in-flight batching continuous or iteration-level batching. Implementations differ; these terms do not describe opposing principles.

What is LLM throughput and latency?

Throughput is how many tokens or requests an LLM can process per second; latency is the time taken to generate responses.

How to reduce latency in LLMs?

By optimizing batching strategies, using efficient hardware, reducing input sequence length, and leveraging caching.

What is the biggest problem with LLM?

The main constraint depends on the application. Evaluate response quality, factual errors, security, latency, memory, and cost against defined acceptance criteria.

What is the throughput of an LLM?

It is the number of tokens or requests an LLM can process per second under given conditions.

How to test LLM throughput?

By measuring tokens generated over time under controlled loads and concurrency.

Is LLM CPU or GPU intensive?

It depends on the engine and deployment. GPUs are common for matrix operations, but CPU and hybrid inference are also supported by some engines. Tokenization and other processing may remain on the CPU.

How to increase LLM throughput?

By batching requests, using optimized inference engines, and deploying on powerful GPUs.

What GPU to use for LLM?

GPUs with high memory and memory bandwidth like Nvidia RTX 4090 or A100 are commonly used.

Does anything LLM use GPU?

GPU support depends on the engine and configuration. CPU-only and hybrid execution are also possible; verify backend support instead of assuming every LLM deployment requires a GPU.

Do I need a GPU to run LLM locally?

For large models, a GPU is recommended; small models can run on CPUs but with reduced performance.

Is RTX 4090 good for LLM?

An RTX 4090 can serve workloads that fit its memory and runtime support. It is a hardware example here, not a current Compute offer: Hivenet has retired its RTX 4090 fleet.

‍