llama-3-3-70b-gpu-requirements

Llama 3.3 70B is too large for a single 32GB RTX 5090.

At BF16, its 70 billion parameters require roughly 140GB for model weights alone. At FP8, that falls to roughly 70GB. A 4-bit representation starts around 35GB, although real quantized checkpoints need more than that simple calculation.

That makes the hardware decision clearer than it first appears:

‍

‍

There is an important qualification behind that table:

VRAM across several GPUs is not automatically pooled into one giant memory space.

The inference engine has to split, or shard, the model across those GPUs. With vLLM, one common way to do that is tensor parallelism.

For a 70B model, that difference between “I have 64GB across two cards” and “my software can actually distribute the model across two cards” is the difference between a useful server and an out-of-memory error.

If you want the single-GPU side of the calculation first, our RTX 5090 VRAM guide explains why weight memory and usable inference memory are different things.

What is Llama 3.3 70B?

Llama 3.3 70B Instruct is Meta's 70-billion-parameter instruction-tuned text model.

Meta released it in December 2024 with a 128K context window, Grouped-Query Attention, multilingual support, and text and code output. It is an autoregressive transformer trained and post-trained for conversational use. The official model card lists English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai as supported languages.

The model is available as:

meta-llama/Llama-3.3-70B-Instruct

through the official Meta model repository.

Its 70B size is the part that matters for hardware.

Unlike a Mixture-of-Experts model such as full DeepSeek-R1, Llama 3.3 70B is a dense model. You cannot look at a much smaller “active parameter” number and use that as the memory requirement.

The model has roughly 70 billion parameters, and those weights have to live somewhere.

How much VRAM does Llama 3.3 70B need?

Start with the same useful approximation we have used elsewhere in this series:

parameters × bytes per parameter = approximate weight memory

For 70 billion parameters:

BF16:
70B × 2 bytes ≈ 140GB

8-bit / FP8:
70B × 1 byte ≈ 70GB

4-bit:
70B × 0.5 bytes ≈ 35GB

That gives us:

The final row is important.

‍

‍

Real quantization is not simply 70 billion perfect four-bit values packed together with no other data.

Quantized checkpoints can include:

• scaling factors
• metadata
• higher-precision components
• an unquantized output head
• tokenizer and model configuration data

Then the inference server still needs GPU memory for:

• KV cache
• CUDA kernels
• temporary buffers
• active sequences
• batching
• framework overhead

NVIDIA's current Llama 3.3 70B NVFP4 checkpoint, for example, is about 42.7GB on disk rather than the theoretical 35GB. NVIDIA reports that its FP4 optimization reduces GPU memory requirements by approximately 3.3× compared with the 16-bit model.

That is small enough to make two 32GB RTX 5090s a realistic starting point.

One is still too small.

Why 70B is different from the models we have already covered

A quantized 14B or 27B model can leave substantial working memory on one RTX 5090.

A 70B model crosses a useful architectural boundary.

Even aggressively quantized weights consume more than one card's memory. So instead of asking:

Which precision lets this model fit on my GPU?

you now have to ask:

How should I distribute this model across several GPUs?

That is why 70B workloads are where tensor parallelism stops being an optional optimization and becomes part of the deployment design.

This is also why our DeepSeek-R1 model-size guide sends 70B-class readers here rather than pretending the same single-GPU advice applies at every parameter count.

What is tensor parallelism?

Tensor parallelism splits individual model operations across several GPUs.

Very roughly, instead of putting an entire large layer on GPU 0 and the next large layer on GPU 1, tensor parallelism divides the tensors involved in those operations so several GPUs work on them together.

For vLLM, you configure this with:

--tensor-parallel-size

If you have two GPUs:

--tensor-parallel-size 2

For four:

--tensor-parallel-size 4

For eight:

--tensor-parallel-size 8

vLLM's current documentation uses Llama 3.3 70B itself as an example of a model deployed with tensor parallelism.

The benefit is straightforward: the model no longer has to fit inside one GPU's VRAM.

The cost is equally real: GPUs now have to communicate during inference.

That communication becomes part of performance.

Two RTX 5090s are the practical FP4 starting point

Two RTX 5090 cloud GPUs provide:

2 × 32GB = 64GB aggregate VRAM

NVIDIA's current quantized Llama 3.3 70B NVFP4 checkpoint is about 42.7GB, and NVIDIA reports approximately a 3.3× reduction in GPU-memory requirements versus the 16-bit model.

That gives two 5090s enough aggregate capacity for the checkpoint and a meaningful amount of additional memory.

The useful qualifier is starting point.

A two-GPU deployment does not mean you should immediately configure the model's full 128K context and expect large request batches.

The remaining memory still has to accommodate KV cache and serving overhead.

For interactive or moderate-context inference, two 5090s and an FP4-class checkpoint are a reasonable place to begin.

For a large production context budget or higher concurrency, four GPUs give you much more room.

Four RTX 5090s are a useful FP8 configuration

Four RTX 5090s provide:

4 × 32GB = 128GB aggregate VRAM

That is comfortably above the approximate 70GB weight requirement of an FP8 70B model.

NVIDIA publishes an official:

nvidia/Llama-3.3-70B-Instruct-FP8

checkpoint built with NVIDIA Model Optimizer.

NVIDIA says the FP8 version reduces GPU-memory requirements by about 50% compared with the 16-bit model. The checkpoint supports vLLM, SGLang, and TensorRT-LLM, including on NVIDIA Blackwell GPUs.

This is an attractive configuration if you care about retaining more numerical precision while still avoiding the memory footprint of BF16.

Four GPUs also leave considerably more aggregate memory for KV cache than trying to force an FP8 model into the smallest possible configuration.

What about BF16?

The weights alone need around 140GB.

Four RTX 5090s provide 128GB.

So four cards are already too small before the server starts doing inference.

The next standard Hivenet configuration is eight GPUs:

8 × 32GB = 256GB aggregate VRAM

That gives enough capacity for BF16 weights plus inference state.

This is the clearest example of why precision has such a large effect on infrastructure:

Those are sensible planning configurations, not universal performance guarantees.

‍

‍

The context length, concurrency target, inference engine, quantization format, and workload still decide whether a specific deployment works well.

Should you use FP4, FP8, or BF16?

Start with the quality your application needs rather than the precision name.

FP4/NVFP4 gives you the smallest memory footprint and makes 70B inference much cheaper to place on Blackwell hardware.

FP8 uses more memory but preserves more numerical precision and remains substantially smaller than BF16.

BF16 avoids weight quantization but demands much more hardware.

NVIDIA's published evaluation for its Llama 3.3 70B FP8 checkpoint shows very small differences from the BF16 reference on the benchmarks it reports. For example, its MMLU score moves from 83.3 at BF16 to 83.2 at FP8.

That is useful evidence.

It is still not your evaluation.

If your model exists to produce valid code, classify insurance documents, follow a specific support policy, or answer technical questions from a particular domain, test those things.

Precision should be a quality-and-cost decision based on your workload.

Step 1: choose the deployment you actually need

For this guide, we will use:

• 2 × RTX 5090
• NVIDIA's FP4/NVFP4 Llama 3.3 70B checkpoint
• vLLM
• tensor parallelism across both GPUs
• a deliberately smaller initial context limit
• an OpenAI-compatible API

This is the lowest-cost practical configuration in our current RTX 5090 fleet for this model class.

If you specifically want FP8, choose four GPUs and change the model checkpoint accordingly.

Create the instance through Compute with Hivenet.

Hivenet supports RTX 5090 instances with one, two, four, or eight GPUs.

Step 2: verify both GPUs

Connect to the instance and run:

nvidia-smi

You should see both RTX 5090s.

A quick Python check is also useful:

python - <<'PY'
import torch

print("CUDA available:", torch.cuda.is_available())
print("GPU count:", torch.cuda.device_count())

for i in range(torch.cuda.device_count()):
   print(i, torch.cuda.get_device_name(i))
PY

For the two-GPU configuration, you want:

GPU count: 2

Do this before debugging vLLM.

Tensor parallelism cannot distribute work across a GPU that the operating system does not see.

Step 3: use a current vLLM build

vLLM now maintains an official Llama 3.3 70B recipe for NVIDIA Blackwell and Hopper hardware, including FP8 and NVFP4 deployment.

That is a better source of truth than an old blog command copied from before Blackwell support existed.

If you already use Hivenet's vLLM workflow, see the Hivenet vLLM guide.

For a manual environment, use a current vLLM release compatible with the checkpoint and CUDA stack.

For example:

python3 -m venv ~/llama70b-env
source ~/llama70b-env/bin/activate

pip install --upgrade pip
pip install --upgrade vllm

For production, pin a tested version after validating the deployment rather than allowing every restart to pull a different dependency set.

Step 4: serve the quantized model across two GPUs

vLLM's current Llama 3.3 recipe provides NVIDIA's Blackwell FP4 checkpoint as:

nvidia/Llama-3.3-70B-Instruct-FP4

Start with a conservative context budget:

vllm serve nvidia/Llama-3.3-70B-Instruct-FP4 \
 --tensor-parallel-size 2 \
 --max-model-len 32768 \
 --kv-cache-dtype fp8 \
 --port 8000

The important setting is:

--tensor-parallel-size 2

That tells vLLM to shard the model across the two GPUs.

The server exposes an OpenAI-compatible API at:

http://localhost:8000/v1

If your environment uses the NVFP4 repository name rather than the shorter FP4 alias, use the exact current checkpoint name documented by NVIDIA and vLLM.

Quantized model formats evolve. Do not rename a checkpoint in a deployment script because two names look equivalent.

Why start with a 32K context?

Llama 3.3 supports 128K context.

That is a model capability, not an instruction to reserve 128K worth of serving memory on every deployment.

vLLM's own Llama 3.3 performance guidance recommends lowering max-model-len when your real input and output lengths are shorter than the model maximum.

That matters because the KV cache consumes memory.

If your application normally receives:

• 4K input tokens
• and produces 1K output tokens

then configuring every request around a theoretical 128K maximum can waste memory that could instead support more concurrent requests.

A 32K starting point already leaves substantial room for long documents and conversations.

Measure first.

Increase it when the workload proves that you need more.

Step 5: test the API

Install the OpenAI client:

pip install --upgrade openai

Create:

test_llama.py

with:

from openai import OpenAI

client = OpenAI(
   base_url="http://localhost:8000/v1",
   api_key="EMPTY",
)

response = client.chat.completions.create(
   model="nvidia/Llama-3.3-70B-Instruct-FP4",
   messages=[
       {
           "role": "user",
           "content": (
               "Explain the difference between GPU VRAM "
               "and aggregate VRAM across a tensor-parallel deployment."
           ),
       }
   ],
   max_tokens=500,
   temperature=0.2,
)

print(response.choices[0].message.content)

Run:

python test_llama.py

At this point you have a working 70B endpoint.

The next job is not adding more flags.

It is measuring whether the deployment behaves properly under the requests you intend to send.

Step 6: monitor both GPUs

While sending requests, run:

watch -n 1 nvidia-smi

You should see memory allocated on both GPUs.

The allocation will not necessarily be numerically identical at every moment, but a tensor-parallel deployment should clearly be using both cards.

vLLM also reports model loading and KV-cache information in its startup logs.

Watch for:

• GPU memory utilization
• KV-cache capacity
• model-load failures
• NCCL communication errors
• out-of-memory errors
• maximum concurrency estimates

The server starting successfully is the beginning of deployment testing, not the end.

Why GPU communication matters

Once a model is split across several GPUs, those GPUs have to exchange intermediate results.

That creates communication overhead.

This is one reason doubling the number of GPUs does not automatically halve latency.

The relationship depends on:

• model architecture
• tensor-parallel degree
• interconnect
• batch size
• sequence length
• serving engine
• GPU utilization

Hivenet has measured NCCL AllReduce behavior on a single host with 8 × RTX 5090 GPUs as part of our GPU VM and bare-metal benchmark work.

That kind of measurement matters more for 70B inference than it does for a model living entirely on one card.

With one GPU, there is nothing to coordinate.

With eight, communication is part of the workload.

More GPUs can lower latency and lower efficiency at the same time

This sounds contradictory until you separate two metrics.

Latency asks how long one user's request takes.

Throughput asks how much work the system completes over time.

vLLM's current Llama 3.3 70B guidance explains that increasing tensor parallelism can improve per-user latency while reducing throughput per GPU at the same batch size.

That makes sense.

More GPUs collaborate on one model execution, so an individual request may finish sooner.

But more hardware is now involved in producing that request, and communication between GPUs adds overhead.

The configuration you want therefore depends on the product.

A private assistant used by ten people has different requirements from an API serving thousands of concurrent requests.

Two GPUs or four for the FP4 model?

Two GPUs are the economical starting point.

Four may still be better for your service.

A four-GPU FP4 deployment gives each GPU a smaller portion of the model and leaves more memory for cache. It can also improve single-request latency under the right conditions.

The price is twice as much GPU capacity.

The question becomes:

Does the additional throughput, context capacity, or latency improvement justify the extra €1.50 per running hour?

That is something you can benchmark directly.

Do not choose four GPUs because 70B sounds large.

Choose four because your two-GPU deployment failed a requirement you can name.

How much system RAM does Llama 3.3 70B need?

System RAM and GPU VRAM solve different problems.

For a GPU-resident vLLM deployment, the model weights should live primarily in GPU memory.

System RAM still supports:

• Linux
• Python
• vLLM
• model downloads and loading
• filesystem cache
• tokenization
• application processes
• monitoring
• other services

If you use CPU offloading, system RAM can also hold part of the model.

That can let you run a configuration that does not fit entirely in VRAM.

It will usually be slower because those weights now have to travel between CPU memory and the GPU.

Adding RAM is therefore not equivalent to adding VRAM.

If performance matters, treat CPU offloading as a deliberate compromise rather than free additional model capacity.

Can Llama 3.3 70B run on one GPU?

Yes, if that one GPU has enough memory.

The problem is not “70B requires multiple GPUs” as a law of nature.

The problem is that a 32GB RTX 5090 does not have enough memory for a useful GPU-resident 70B checkpoint.

A single accelerator with much more memory can run configurations that require several 5090s.

NVIDIA's current vLLM recipe, for example, demonstrates an FP4 Llama 3.3 70B deployment on a single B200-class Blackwell accelerator.

That hardware has a completely different memory budget.

Always translate “runs on one GPU” into:

Which GPU, with how much VRAM, using which precision?

Without those details, the statement tells you little.

Can you run Llama 3.3 70B with Ollama?

Yes. Ollama publishes Llama 3.3 70B in its model library.

That does not remove the memory requirement.

Ollama can distribute a model across available GPUs and can also place some layers in CPU memory when the model does not fit completely in VRAM.

That makes it convenient for experimentation.

For production API serving where throughput, batching, tensor-parallel configuration, and observability matter, I would generally choose vLLM.

We'll cover Ollama's GPU behavior separately in our planned Ollama GPU and cloud deployment guide.

The useful distinction is not “Ollama for beginners, vLLM for experts.”

They are different serving tools with different priorities.

How does Llama 3.3 70B compare with DeepSeek-R1 70B?

DeepSeek-R1-Distill-Llama-70B uses Llama 3.3 70B Instruct as its base model.

DeepSeek then fine-tuned it on reasoning data produced by full DeepSeek-R1.

That means the two models have approximately the same parameter-count hardware class.

A 70B DeepSeek distill does not magically become a 32B workload because it came from R1.

The same broad memory math applies:

~140GB at BF16
~70GB at 8-bit
~35GB theoretical at 4-bit

Their behavior is different.

Their hardware class is similar.

See our DeepSeek-R1 model sizes and VRAM guide for the rest of that family.

What does Llama 3.3 70B cost on RTX 5090s?

Hivenet currently lists RTX 5090 Compute from €0.75 per GPU-hour, with per-second billing.

That gives:

So an FP4 two-GPU experiment running for two hours costs about:

‍

‍

2 hours × €1.50 = €3.00

A four-GPU FP8 server left running continuously for 24 hours costs:

24 × €3.00 = €72.00

That is why deployment economics cannot stop at hourly price.

For a production endpoint you need to measure how much useful work each configuration performs.

Relevant metrics include:

• time to first token
• tokens per second
• total token throughput
• requests per second
• concurrency
• p50 and p99 latency
• GPU utilization
• cost per million useful output tokens

vLLM includes vllm bench serve specifically for measuring serving performance rather than guessing from specifications.

How to benchmark the deployment

A basic vLLM benchmark can look like:

vllm bench serve \
 --host 127.0.0.1 \
 --port 8000 \
 --model nvidia/Llama-3.3-70B-Instruct-FP4 \
 --dataset-name random \
 --random-input-len 1024 \
 --random-output-len 512 \
 --num-prompts 500

For internal comparison, keep the workload constant.

If you benchmark two GPUs with 1,024-token prompts and four GPUs with 8,000-token prompts, you have mostly measured two different workloads.

Compare:

• the same model
• same checkpoint
• same context
• same prompt distribution
• same output length
• same concurrency

Then change one infrastructure variable.

That is how you find out whether another two GPUs actually buy you something.

Do you need all 128K of context?

Probably not for every request.

Meta gives Llama 3.3 a 128K context window.

That is valuable when you genuinely need long documents, long conversation history, or large amounts of retrieved context.

It also has a cost.

Long context increases KV-cache use, prefill work, and time before generation begins.

A production service that sends 100K tokens with every request because the model supports 128K will pay for those tokens in latency and capacity.

Use long context where the task requires it.

For document-heavy applications, retrieval can also be more efficient than copying an entire corpus into every prompt. Our RAG with Hivenet page covers that infrastructure path.

Does Llama 3.3 70B make sense in 2026?

It can.

Newer models existing does not make an older model useless.

The question is whether Llama 3.3 70B still provides the combination you need:

• known behavior
• broad tool support
• multilingual capability
• mature serving support
• established quantized checkpoints
• a license your organization can work with
• performance that passes your evaluation

Its age can even be useful operationally.

Serving engines, quantizers, deployment recipes, and tooling have had time to stabilize around the model.

That matters when you care more about running a dependable workload than testing the newest checkpoint published this week.

Do not choose it because “70B” sounds powerful.

Do not reject it because something newer exists.

Evaluate the model against the job.

Check the Llama license before deployment

Llama 3.3 is not released under Apache 2.0 or MIT.

Meta distributes it under the Llama 3.3 Community License.

The license allows commercial and research use under its terms, but it includes requirements around redistribution, attribution, derivative models, and certain large-scale commercial use.

The official Llama 3.3 model card links directly to the current agreement.

Read it before distributing the model or building a derivative.

“Open weights” does not mean “there are no license conditions.”

When should you choose a smaller model?

A 70B model is a poor infrastructure choice when a 14B or 27B model passes the same product evaluation.

Smaller models generally give you:

• lower latency
• higher concurrency
• lower cost
• simpler deployment
• larger context headroom per GPU
• fewer moving parts

Our Qwen3.6-27B deployment guide shows how a quantized 27B model can fit on one RTX 5090.

That is a radically simpler deployment than a tensor-parallel 70B server.

Use the additional 43 billion parameters because they improve the result you need.

Otherwise, you are paying to move more weights around.

When should you choose Llama 3.3 70B?

A 70B deployment becomes easier to justify when:

• a smaller model consistently misses your quality threshold
• multilingual performance matters
• the model is already validated within your organization
• you need a mature open-weight model with broad runtime support
• you want control over the full serving stack
• the workload can keep several GPUs usefully occupied

If your service receives five requests per day, a permanently running four-GPU server is difficult to justify.

If a team is sending sustained inference traffic through the model, the calculation changes.

Infrastructure is part of model selection.

Self-managed 70B inference or a managed endpoint?

This tutorial assumes you want control over the serving layer.

Compute with Hivenet lets you manage:

• the exact model checkpoint
• precision
• GPU count
• vLLM version
• tensor parallelism
• context limits
• batching
• API configuration

That control is useful when you need it.

If what you actually want is an OpenAI-compatible endpoint without operating a multi-GPU inference server, Hivenet Inference API is the more appropriate path.

A managed endpoint and GPU rental solve different problems.

Do not turn server administration into a requirement if server administration is not part of the value you need.

Llama 3.3 70B FAQ

How many parameters does Llama 3.3 70B have?

Llama 3.3 70B has approximately 70 billion parameters. It is a dense autoregressive transformer rather than a Mixture-of-Experts model.

How much VRAM does Llama 3.3 70B need?

The model weights require roughly 140GB at BF16, 70GB at FP8, and a theoretical 35GB at 4-bit. Real inference needs additional VRAM for quantization metadata, KV cache, temporary buffers, and active requests.

Can Llama 3.3 70B run on one RTX 5090?

No, not as a normal fully GPU-resident deployment. The RTX 5090 has 32GB VRAM, while even the theoretical 4-bit weight requirement is about 35GB.

Can Llama 3.3 70B run on two RTX 5090s?

Yes, a suitable FP4/NVFP4 checkpoint can be sharded across two 32GB RTX 5090s. Tensor parallelism is required to distribute the model rather than treating the cards as unrelated memory.

How many RTX 5090s do I need for Llama 3.3 70B?

A practical planning guide is two GPUs for a suitable FP4/NVFP4 deployment, four for FP8 with useful headroom, and eight for BF16. Exact requirements depend on the checkpoint, context, concurrency, and inference engine.

Does adding two RTX 5090s create a 64GB GPU?

No. It gives the server 64GB of aggregate VRAM across two separate GPUs. The inference framework must explicitly shard the model between them.

What is tensor parallelism?

Tensor parallelism splits model tensor operations across several GPUs so they work together on the same inference request. In vLLM it is configured with --tensor-parallel-size.

Does Llama 3.3 70B support 128K context?

Yes. Meta lists a 128K context window for Llama 3.3 70B. Serving the full context requires substantially more KV-cache capacity than using a smaller context limit.

Is Llama 3.3 70B multilingual?

Yes. Meta explicitly lists English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai as supported languages.

Is Llama 3.3 70B open source?

The model weights are publicly available under Meta's Llama 3.3 Community License. “Open-weight” is the clearer term because use and redistribution remain subject to license conditions.

Can vLLM serve Llama 3.3 70B?

Yes. vLLM supports the model and maintains a current Llama 3.3 70B deployment recipe for NVIDIA Blackwell and Hopper GPUs, including FP8 and NVFP4 configurations.

Is FP4 always better than FP8 for Llama 70B?

No. FP4 saves more memory and can reduce hardware requirements, while FP8 retains more precision. Evaluate the relevant checkpoint on your workload before deciding.

Should I use two or four GPUs?

Start with two for a suitable FP4 checkpoint if cost matters and your context and concurrency requirements are moderate. Move to four when measurements show you need more cache capacity, lower latency, more concurrency, or FP8 precision.

How much does a two-GPU Llama 3.3 70B server cost on Hivenet?

At the current published RTX 5090 rate of €0.75 per GPU-hour, two GPUs cost €1.50 per running hour. Compute uses per-second billing.

‍