RTX 5090 VRAM for AI: what fits in 32GB?

The RTX 5090 has 32GB of GDDR7 VRAM. For AI workloads, that is enough for 7B and 8B language models at high precision, many models in the 20B–30B range with quantization, and demanding image-generation workflows. A 70B model is generally too large for one 32GB GPU, even at 4-bit precision.

That simple answer hides the part that matters in practice. Model weights aren't the only thing competing for GPU memory. Context length, KV cache, concurrent requests, inference-engine overhead, and the precision you use can turn a model that looks like a comfortable fit on paper into one that runs out of memory under a real workload.

If you're choosing an RTX 5090 cloud GPU for AI, the useful question is therefore not just how many parameters it can load. It's how much useful headroom remains once the model is running.

How much VRAM does the RTX 5090 have?

The desktop NVIDIA GeForce RTX 5090 has 32GB of GDDR7 VRAM with 1,792 GB/s of memory bandwidth. It is based on NVIDIA's Blackwell architecture and supports lower-precision formats designed to reduce memory use and accelerate AI workloads.

This article is about the desktop RTX 5090 used for cloud GPU workloads, including the RTX 5090s available through Compute with Hivenet. Laptop GPUs with similar names have different specifications and shouldn't be treated as equivalent hardware.

For AI, VRAM capacity and memory bandwidth solve different problems.

VRAM capacity determines what fits. The model weights, cache, runtime, and working data all need somewhere to live.

Memory bandwidth affects how quickly data can move. High bandwidth can improve performance for memory-heavy workloads, but it cannot compensate for a model that is simply too large for the card.

A fast 32GB GPU is still a 32GB GPU.

What AI model sizes fit in 32GB of VRAM?

You can get a useful first estimate from a simple calculation:

number of parameters × bytes per parameter = approximate memory for the weights

For common precisions, that gives us roughly:

These are weights-only estimates. They are useful for ruling configurations in or out, but they are not promises about actual VRAM consumption.

A 32B model at 8-bit is a good example. The arithmetic gives you about 32GB of weights. That does not make it a sensible fit for a 32GB GPU, because there would be no meaningful room left for the serving engine, cache, temporary allocations, or the workload itself.

The right target is rarely a model that just squeezes into memory.

Why model weights don't tell you the whole story

An LLM needs memory for its parameters, but inference also creates working state.

The biggest variable is often the KV cache, which stores information from tokens that have already been processed so the model doesn't have to recompute them at every generation step.

Longer context means a larger cache. More concurrent requests mean more cache again.

Inference engines such as vLLM therefore have to divide the available GPU memory between model execution and KV cache rather than treating the full 32GB as a bucket for weights. If you're using Hivenet, our vLLM guide covers the serving setup itself.

This is why two people can run the same model on the same GPU and report different memory limits. One may be testing a short single-user prompt. The other may be serving long conversations to multiple users.

Both results can be correct.

7B and 8B models fit comfortably

Seven- and eight-billion-parameter models are the easy case.

At BF16, their weights occupy roughly 14GB to 16GB. That leaves a useful amount of the RTX 5090's 32GB available for context, cache, and the inference runtime.

You can still quantize them. Doing so can free memory for higher concurrency or longer contexts and may improve throughput on hardware that handles the lower precision efficiently.

But you generally don't need aggressive quantization simply to make an 8B model load on the GPU.

That makes this size range useful when latency and predictable serving behavior matter more than running the largest model the hardware can technically hold.

14B models fit, but BF16 leaves little headroom

A 14B model needs roughly 28GB for BF16 weights.

That is below 32GB, but the difference isn't enough to treat the remaining memory as comfortable working space. Once cache and runtime overhead enter the calculation, the limits arrive quickly.

At 8-bit, the rough weight requirement falls to around 14GB. That gives the same GPU much more room for context and concurrent requests.

This illustrates an important difference between can run and runs well.

If your application needs short prompts and single-user experimentation, a tight configuration may be acceptable. If you're building an API that has to remain responsive under simultaneous requests, leaving memory headroom is part of the infrastructure design.

27B and 32B models are where quantization changes the calculation

Without quantization, this model range is too large for a single RTX 5090.

A 27B model needs roughly 54GB for BF16 weights. A 32B model comes out around 64GB.

At 4-bit, the basic arithmetic drops those numbers to approximately 13.5GB and 16GB.

Real quantized checkpoints require more than that simplistic calculation because quantization formats can include scales, metadata, and selected components kept at higher precision. Even so, the memory reduction is large enough to move these models from impossible to practical on a 32GB GPU.

This is also where Blackwell-specific quantization becomes interesting.

For example, our HivenetQuant Qwen3.6-27B-NVFP4 uses NVFP4 W4A4 quantization for the MLP while keeping the attention and recurrent DeltaNet layers at FP8. The layer split is chosen according to weight sensitivity rather than forcing every part of the model into the same precision.

We'll go through the actual setup and performance tradeoffs in our Qwen3.6-27B cloud GPU guide.

The wider point matters more than any single format: quantization is a model-specific engineering choice, not a magic compression button.

A good quantized checkpoint can save a large amount of memory with little practical loss for a given workload. A poor one can save memory at the expense of quality you actually need.

Test the model you plan to deploy.

Can an RTX 5090 run a 70B model?

Not comfortably on one GPU.

At 4-bit, 70 billion parameters require roughly 35GB for weights before accounting for the quantization format or any runtime memory. That already exceeds the RTX 5090's 32GB capacity.

CPU offloading can make some oversized models run by keeping part of the model in system RAM, but that changes the performance characteristics considerably. Data has to move between CPU memory and GPU memory, and the GPU can no longer work with the entire model locally.

For sustained inference, multiple GPUs are usually the cleaner answer.

Compute with Hivenet supports multi-GPU configurations, so larger models can be distributed across several GPUs rather than forced onto a card that doesn't have enough memory.

Our planned Llama 3.3 70B GPU requirements guide will go into quantization, tensor parallelism, and the practical difference between fitting a 70B model and serving it efficiently.

Context length can make a model that fits stop fitting

Maximum context length is easy to read from a model card. The harder question is whether your hardware can serve it efficiently.

The KV cache grows as the amount of active context increases. Its actual size depends on the model architecture, cache precision, batch size, sequence length, and inference engine.

Suppose a quantized 27B model leaves plenty of memory after its weights are loaded. With short prompts and one request at a time, the setup can look comfortable.

Increase the context substantially and serve several users at once, and the cache can consume much more of the remaining VRAM.

That means a model's advertised maximum context window and the context window that makes sense on a given GPU are different things.

Plan around the workload you expect to serve rather than the largest number printed on the model page.

DeepSeek makes the model-size problem particularly visible

"Can I run DeepSeek-R1?" sounds like a hardware question, but it is incomplete.

DeepSeek-R1 exists alongside much smaller distilled models, and those configurations have radically different memory requirements. Treating "DeepSeek-R1" as one fixed-size workload is how people end up with misleading GPU recommendations.

On a 32GB RTX 5090, the sensible question is which distilled model and precision suit the workload.

We'll map those choices separately in our DeepSeek-R1 model size and memory requirements guide.

The same discipline applies to any model family. Use the exact checkpoint, precision, context requirement, and serving setup when calculating memory.

Is 32GB enough for Stable Diffusion and FLUX?

For many image-generation workloads, yes, but the memory calculation looks different from LLM inference.

Image models don't map as neatly onto the parameter-count table above because the pipeline can include several large components and memory use changes with resolution, batch size, upscaling, ControlNet-style additions, and the interface or workflow you're running.

NVIDIA reports that FLUX.1 [dev] requires more than 23GB of VRAM in ordinary FP16 operation. That is an interesting threshold because it puts the workload beyond the comfortable range of a 24GB GPU while leaving some room on a 32GB RTX 5090.

Blackwell's FP4 support can reduce memory requirements further for compatible workflows.

We already cover the broader hardware question in our Stable Diffusion GPU and VRAM requirements guide. We'll also publish a hands-on FLUX.1 [dev] cloud GPU guide for the actual workflow.

Again, fitting the base model is only part of the job. A workflow that adds larger resolutions, multiple models, upscaling, or additional conditioning can consume considerably more memory.

Fine-tuning needs more VRAM than inference

A model that fits comfortably for inference may be far too large for full-parameter training on the same GPU.

Training has to store more than the model weights. Depending on the method, memory is also consumed by gradients, optimizer states, activations, batches, and other training data.

That is why full fine-tuning becomes expensive in VRAM so quickly.

Parameter-efficient approaches such as LoRA and QLoRA reduce the requirement by training a much smaller set of parameters while leaving most of the base model unchanged.

This makes them a much better match for single-GPU experimentation.

Our training and fine-tuning page covers the infrastructure side. The upcoming LoRA fine-tuning guide will walk through the actual memory tradeoffs and training process.

How much headroom should you leave?

There is no percentage that works for every model and every inference engine.

A better rule is to avoid selecting a checkpoint whose weights consume nearly all available VRAM before your workload begins.

The amount of headroom you need depends on what you're doing:

• Short, single-user inference can tolerate a tighter configuration.
• Long context needs more cache.
• Higher concurrency needs more memory for active requests.
• Image workflows need room for the rest of the pipeline.
• Fine-tuning has a completely different memory profile from inference.

Start with the expected workload, then choose the largest model that fits it comfortably.

If you're comparing configurations, our Hivenet benchmarks collect the performance results we publish rather than asking you to infer performance from hardware specifications alone.

So, is 32GB of VRAM enough for AI?

For a large share of current AI work, yes.

The RTX 5090's 32GB of VRAM can handle smaller LLMs at high precision, many 20B–30B models after suitable quantization, demanding image-generation pipelines, and parameter-efficient experimentation.

It doesn't make memory planning disappear.

Large 70B-class models, full-parameter fine-tuning, extreme context lengths, and high-concurrency serving can all push beyond one card.

That's the useful boundary to understand.

The RTX 5090 isn't interesting because 32GB can somehow run everything. It is interesting because 32GB, high memory bandwidth, and Blackwell's low-precision support cover a wide range of practical AI workloads on one GPU. When a workload genuinely needs more memory, using multiple GPUs is a better engineering decision than pretending the limit isn't there.

RTX 5090 VRAM FAQ

How much VRAM does the RTX 5090 have?

The desktop NVIDIA GeForce RTX 5090 has 32GB of GDDR7 VRAM.

Is 32GB of VRAM enough for AI?

It is enough for many AI workloads. Smaller LLMs can run at higher precision, while many models in the 20B–30B range become practical with quantization. Larger 70B-class models generally require more than one 32GB GPU.

Can an RTX 5090 run a 32B LLM?

Yes, with suitable quantization. A 32B model needs roughly 64GB for BF16 weights alone, while a 4-bit representation starts around 16GB before quantization overhead, cache, and runtime memory.

Can an RTX 5090 run a 70B model?

A single RTX 5090 is generally too small. Even a simple 4-bit weight estimate for a 70B model is around 35GB before runtime overhead. Multi-GPU inference is the more practical approach.

How much VRAM does a 14B model need?

A 14B model requires roughly 28GB for BF16 weights, 14GB at 8-bit, or 7GB at 4-bit before additional memory use. BF16 can therefore be tight on a 32GB GPU, while lower precision leaves considerably more working room.

Does quantization reduce VRAM use?

Yes. Quantization represents some or all model weights using fewer bits, which reduces memory use. The exact saving and quality impact depend on the quantization format and the model.

Does a longer context use more VRAM?

Yes. Longer active context increases the KV-cache requirement during LLM inference. Concurrency, model architecture, cache precision, and the serving engine also affect how much memory is needed.

Is the RTX 5090 good for LoRA fine-tuning?

It can be a useful GPU for parameter-efficient fine-tuning, depending on the model and training configuration. LoRA and QLoRA require far less memory than full-parameter fine-tuning because they update only a small part of the model.