Which GPU should you use for LLM inference

Memory is an early constraint when choosing inference hardware. Allow room for the weights, KV cache, and engine workspace, then measure latency and throughput under realistic load. Fitting in VRAM alone does not guarantee stable response times.

The underlying GPU architecture plays a crucial role in determining inference performance and the suitability of hardware for LLM deployment, as architectural differences impact efficiency and scalability for large models.

On Compute, you can launch a vLLM server on Compute using an available RTX 5090 preset. Check the console for the current GPU count, regional capacity, and price before choosing an instance.

A quick decision path

1. Pick the smallest model that solves the task. Try 7B before 13B. Use evals, not vibes.
2. Estimate context and outputs honestly. Long chats and big prompts eat memory.
3. Target concurrency. How many users at once with acceptable TTFT/TPS?
4. Choose VRAM to fit model + cache + batch. Consider how much GPU memory and how much memory are required for your specific model size and batch requirements—larger models and higher batch sizes need more memory. If close to the limit, step up a tier, use quantization, or consider lower precision formats (like FP8 or int8) as a trade-off to optimize memory usage and throughput.
5. Decide single vs multi‑GPU. Go multi when one card cannot meet memory or throughput needs. Optimizing multi-GPU setups can improve performance, but there are trade-offs between single and multi-GPU configurations, such as increased complexity and cost.
6. Place the endpoint near users. Regional latency matters more than micro‑optimizations.

Review GPU options for your inference workload.

Check current presets, regional capacity, and pricing before launching. Review the setup requirements for your workload and confirm any support needs with Hivenet.

Try Compute now

Estimate weight memory

These figures describe raw weights only, in decimal GB: parameter count × bits per weight ÷ 8 ÷ 10⁹. They exclude quantization metadata, tensors kept at other precisions, the KV cache, and engine workspace. They are not minimum GPU-memory recommendations.

• 7B, FP16: about 14 GB for weights.
• 7B, INT8: about 7 GB for weights.
• 7B, INT4: about 3.5 GB for weights.
• 13B, FP16: about 26 GB for weights.
• 13B, INT8: about 13 GB for weights.
• 13B, INT4: about 6.5 GB for weights.

Serving LLMs (large language models) efficiently requires careful planning of GPU memory allocation for both the model and its cache, as LLM inference is computationally demanding and benefits from specialized hardware.

Add space for the KV cache and engine. Requirements depend on architecture, precision, context length, and active requests. High reserved VRAM alone does not diagnose a problem. Check preemptions, queues, and errors; the vLLM tuning guide explains the tradeoffs.

Single vs multi‑GPU

Single‑GPU is simpler and often faster for 7B‑class models with moderate context. Start here if you can.

Multi‑GPU helps when the model or context does not fit, or when you need more throughput at the same latency target. Use tensor or pipeline parallelism, and test batch shapes. Parallelism adds communication overhead, so measure with your real prompts. It is crucial to measure real-world performance and communication overhead when optimizing multi-GPU setups.

Consumer vs data‑center parts

Consumer GPUs. RTX 4090 and RTX 5090 cards can serve some quantized or moderate-size models, depending on context and concurrency. Compare measured cost per workload rather than assuming that consumer cards always offer better value. Hardware examples do not establish current Hivenet availability.

Data-center GPUs. A100, H100, and L40S configurations differ in memory, compute, and interconnects. Check the exact card and server. For example, NVIDIA lists the L40S with 48 GB of ECC memory but no NVLink or MIG support.

If you need ECC, partitioning, or a particular interconnect, verify it on the selected configuration. Cost per token depends on the model, engine, and traffic; neither hardware category wins for every workload.

Latency and throughput, briefly

• TTFT. The client-side wait for the first token includes queueing, prefill, and network time. Longer prompts can increase it. Check the measurement boundary before comparing results.
• Tokens per second (TPS). Distinguish one response's output rate from aggregate server throughput. More VRAM may permit more active requests without guaranteeing faster output. Use consistent metric definitions.
• Network placement can add 50–100 ms in a blink; keep endpoints close to users.

Power, thermals, and reliability

Hot cards throttle. Use cases with steady load need good airflow and power headroom. Data‑center parts are built for this; consumer cards can do it with care. Monitor temps and clocks.

Region placement

Place the endpoint where most users are. EU users benefit from France. Middle‑East markets benefit from UAE. Cross‑region calls add latency you cannot optimize away in code.

A budgeting approach you can reuse

1. Estimate requests/day. Benchmark representative prompt and output lengths on the intended instance at your target concurrency and latency.
2. Divide by measured completed requests/second. Use throughput for that same workload and instance configuration to estimate busy instance-seconds/day.
3. Divide by 3,600 for instance-hours/day. Multiply by the full-instance hourly rate, then include idle running time and spare capacity in the billed-hours estimate.
4. Run a sensitivity check. Vary context and max tokens; these swing cost the most.
5. Decide on redundancy. One spare node costs money but saves incidents.

Monitoring that pays off

• TTFT p50/p95 under rising load
• TPS p50/p95 at steady traffic
• GPU memory headroom and cache hit rate
• Thermal throttling events
• Error rates (OOM, timeouts, 5xx)

Quick checklist

• Start with the smallest model that passes evals.
• Choose VRAM with headroom for context and batch.
• Prefer single‑GPU until you must scale out.
• Stream responses and cap max_tokens.
• Place endpoints in the region users live in.
• Watch TTFT/TPS, memory, temps, and errors.

Try Compute today

On Compute, choose an available RTX 5090 preset and verify its GPU count, location, and price in the console. Configure your vLLM server and verify its endpoint before connecting a client.

Final recommendations for choosing the best GPU for LLM inference

Pick GPUs by VRAM first, then speed, then price. Keep endpoints close to users, stream responses, and watch TTFT and memory. Let clean measurements, not spec sheets, drive upgrades.

Ready to test? Launch a vLLM endpoint on Compute, choose your region and preset, and compare TTFT/TPS before you commit to a bigger card.

FAQ

What GPU is enough for a 7B chat model?

A 24 GB card usually works well, especially with int8 or int4 variants and sensible caps. Keep headroom for the cache and batching.

When do I need multi‑GPU?

When the model or context will not fit on one card with headroom, or when you need higher throughput at the same latency target.

Do I need NVLink?

Helpful for very large models and long contexts across multiple GPUs. For 7B–13B with moderate context, you can often stay on a single card.

4090 vs A100 vs H100—how should I think about it?

Compare usable memory, runtime support, precision, interconnects, and measured cost on your workload. Published H100-versus-A100 multipliers describe specific benchmark configurations; they are not universal gains over an RTX 4090 or your existing service.

What changes for long context (32k+)?

Long contexts increase cache demand. Measure the total footprint, then consider more memory, shorter inputs, or supported multi-GPU execution. RAG may limit the context you send, but retrieved passages also consume tokens.

Will quantization let me drop a GPU tier?

Quantization can reduce weight memory, but a smaller GPU is not guaranteed. Check format, engine, and hardware compatibility, then measure quality, memory, and throughput on your tasks.

Do you need GPU for LLM inference?

No. Engines such as llama.cpp support CPU and hybrid execution as well as GPU acceleration. Choose hardware based on available memory and your latency and throughput targets.

What GPU do I need for LLM?

Size the hardware for the model, precision, context, and concurrency. Exactly 13 billion FP16 parameters occupy about 26 decimal GB before cache and workspace, so those weights do not fit entirely in 24 GB. Quantization or splitting the model can change the requirement. Training and fine-tuning need additional memory.

How to choose GPU for inference?

Consider the model size, required VRAM, throughput needs, latency targets, and budget. Start by ensuring the GPU has enough memory for the model and its cache, then evaluate performance factors like CUDA cores, tensor cores, and memory bandwidth. Also, consider single vs multi-GPU setups based on workload scale.

How should I compare an A100 with an RTX 4090?

Use the same model, precision, engine, request lengths, and concurrency. Measure throughput, latency, and cost, and record memory and interconnect details. Specifications alone do not establish which is faster for your service.

What is the difference between H100 and A100 vs RTX 4090?

The H100 and A100 are data-center GPUs optimized for AI workloads with features like higher VRAM, tensor cores, NVLink, and multi-instance GPU support. The RTX 4090 is a consumer GPU with excellent performance and efficiency for smaller models but lacks some enterprise features and large memory pools found in H100/A100. Both the H100 and A100 are ideal for large-scale AI workloads, while the RTX 4090 is more suited for smaller tasks.

Is the Nvidia A100 still relevant?

The A100 may remain suitable for a workload. Compare its usable memory, software compatibility, availability, and measured cost with alternatives rather than judging it by age alone.

Is RTX 4090 still available on Compute?

No. Hivenet's GPU reference states that its RTX 4090 fleet has been retired. That describes Compute availability, not worldwide manufacturing or retail status.

How much VRAM does a 7B model need?

For exactly 7 billion parameters, raw FP16 weights occupy about 14 decimal GB. Total runtime memory also includes the cache and engine. Quantization reduces weight storage but does not eliminate those other requirements.

What GPU will run 7B model?

Parameter count alone does not set a minimum VRAM capacity. Choose the precision, context length, and concurrency, then measure the total memory requirement and performance on candidate hardware.

What GPU do you need for Mistral 7B?

Check the exact Mistral checkpoint, precision, and serving engine. Cache requirements also depend on context and concurrency. Quantized versions can use less memory than FP16 versions; a universal 16 GB minimum would be misleading.

‍