Tensor parallelism explained: how multi-GPU LLM inference works

Tensor parallelism lets several GPUs work together on one copy of a model by splitting the tensors inside its layers across those GPUs.

If a model needs 70GB of weights and each GPU has 32GB of VRAM, tensor parallelism can distribute those weights across several cards rather than requiring one GPU large enough to hold the entire model.

But two 32GB GPUs do not become one 64GB GPU.

Each GPU still has its own memory. The inference framework decides which part of the model each GPU stores, sends the relevant input through those shards, and coordinates the partial results as the model moves from layer to layer.

That communication is the cost of tensor parallelism.

Current vLLM guidance describes tensor parallelism as its most common strategy for large-model inference within a single node. It recommends TP when a model is too large for one GPU or when distributing the weights leaves more per-GPU memory available for KV cache and higher serving throughput.

The practical question is therefore not:

How many GPUs can I add?

It is:

What is the smallest tensor-parallel configuration that fits the model, leaves enough memory for the workload, and still communicates efficiently?

What is tensor parallelism?

A neural-network layer contains large matrices.

A simplified linear operation looks like:

Y = XW

where:

• X is the input
• W is the layer's weight matrix
• Y is the result

If W is too large, tensor parallelism can split it across GPUs.

Conceptually:

              Weight matrix W
       ┌────────────┬────────────┐
       │            │            │
       │   GPU 0    │   GPU 1    │
       │  shard W0  │  shard W1  │
       │            │            │
       └────────────┴────────────┘
             ↑             ↑
             └──── input ──┘

Both GPUs work on the same model layer at the same time.

They each calculate a portion of the result.

Those partial results then have to be combined or redistributed so the next operation can continue.

That is different from assigning:

layers 1–40 → GPU 0
layers 41–80 → GPU 1

which is closer to pipeline parallelism.

Tensor parallelism splits work within layers.

Pipeline parallelism splits work between groups of layers.

vLLM describes the same distinction: TP shards model parameters inside each layer, while pipeline parallelism places different layers on different GPUs and processes them in sequence.

Tensor parallelism does not pool GPU memory

This is the most important misconception to remove.

Suppose you have:

2 × RTX 5090

Each GPU has:

32GB VRAM

The system has:

64GB aggregate VRAM

but there is no single physical:

64GB VRAM address space

that every CUDA operation can use without coordination.

Instead, the framework might place approximately half of a sharded weight matrix on GPU 0 and the other half on GPU 1.

Each device works with its own memory.

The software makes the model distributed.

It does not turn the hardware into one larger GPU.

That distinction matters because any operation needing information from another shard now involves communication.

Why use tensor parallelism?

The first reason is simple:

the model does not fit on one GPU.

Imagine a 70B dense language model.

Very roughly:

BF16:
70B × 2 bytes
≈ 140GB

FP8:
70B × 1 byte
≈ 70GB

4-bit:
70B × 0.5 bytes
≈ 35GB

These are weights-only estimates.

A single RTX 5090 has 32GB VRAM, so even the theoretical 4-bit weight calculation is already outside one card before runtime overhead and KV cache enter the picture. Hivenet's current RTX 5090 instances provide 32GB GDDR7 per GPU and scale from one to eight GPUs in a single instance.

This is the boundary we explored in our Llama 3.3 70B GPU requirements guide.

Tensor parallelism is one way across it.

A simple 70B example

Take a hypothetical 70B BF16 model with approximately:

140GB

of raw weights.

If those weights could be divided perfectly evenly, the theoretical weight share would be:

‍

‍

This does not mean a four-GPU 32GB configuration can automatically serve a 140GB BF16 model.

At TP=4, the raw idealized weight share is already about 35GB per GPU, which exceeds each RTX 5090's 32GB.

At TP=8, the simple weight division leaves much more room:

~17.5GB per GPU

for the sharded weights, before other allocations.

That is why our Llama guide uses eight RTX 5090s as the sensible BF16 planning class rather than adding weight sizes and assuming aggregate VRAM settles the problem.

Real model memory is not divided perfectly

The simple table helps build intuition.

Real deployment is messier.

Not every allocation is necessarily sharded by exactly the same factor.

Depending on the model and runtime, GPU memory can also contain:

• embeddings
• normalization parameters
• output heads
• temporary buffers
• CUDA graphs
• attention workspace
• KV cache
• multimodal components
• quantization metadata
• runtime allocations

So:

model size ÷ number of GPUs

is a useful lower-level estimate for the sharded part of the model.

It is not a complete VRAM forecast.

The only reliable final test is loading the actual checkpoint with the actual serving stack and measuring the workload.

Tensor parallelism can help even when the model already fits

Model fit is not the only reason to shard.

vLLM also recommends tensor parallelism when reducing the weight footprint per GPU gives the server more memory for KV cache, which can increase serving capacity.

Suppose a model fits on one GPU but leaves:

4GB

for cache and runtime state.

A two-way tensor-parallel deployment may reduce the model memory consumed on each card enough to leave substantially more cache space per GPU.

That extra room can support:

• longer active contexts
• more concurrent requests
• larger batches

Whether the additional GPU is worth the cost depends on how much serving capacity that memory buys.

Our KV cache guide explains why active context can consume several gigabytes of VRAM even after the model weights fit.

How KV cache interacts with tensor parallelism

In a conventional tensor-parallel attention implementation, attention heads or KV heads can also be distributed across the tensor-parallel ranks.

That means each GPU may only need the cache associated with the attention state assigned to its shard.

Current vLLM documentation explicitly describes tensor-parallel sharding of KV cache along the head dimension as a way to create more cache capacity when one GPU cannot hold enough active requests.

That is another reason the useful question is not simply:

How much aggregate VRAM do I have?

The model architecture and parallel implementation decide which parts of the inference state are actually distributed.

What does tensor_parallel_size mean in vLLM?

In vLLM, tensor parallelism is configured through:

tensor_parallel_size

or the command-line equivalent:

--tensor-parallel-size

For four GPUs:

vllm serve <model> \
 --tensor-parallel-size 4

vLLM then distributes each supported tensor-parallel model replica across four GPUs. Its current serving documentation uses this exact pattern for multi-GPU inference.

In Python:

from vllm import LLM

llm = LLM(
   model="<model>",
   tensor_parallel_size=4,
)

For a real model, use the deployment instructions associated with the checkpoint rather than treating those three lines as a complete production configuration.

The purpose here is to show what TP = 4 means.

One model replica is spread over four GPUs.

What happens during a tensor-parallel layer?

Transformer implementations commonly use combinations of column-parallel and row-parallel linear layers.

You can think about a large matrix being split vertically:

W = [ W0 | W1 | W2 | W3 ]

Each GPU receives the input and computes:

GPU 0 → XW0
GPU 1 → XW1
GPU 2 → XW2
GPU 3 → XW3

Those pieces represent different parts of the layer output.

Another operation may split a matrix in the complementary direction and eventually require partial results from all GPUs to be reduced or gathered.

The exact communication pattern varies with the layer and framework.

The important part is that tensor parallelism repeatedly creates points where GPUs need to exchange data.

The model is distributed inside the forward pass.

That is why GPU-to-GPU communication performance matters so much.

What is NCCL?

NCCL is NVIDIA's Collective Communications Library.

It implements GPU communication operations used by distributed workloads.

One important operation is AllReduce.

NVIDIA defines AllReduce as taking data supplied by several ranks, applying a reduction such as a sum, and making the combined result available to every participating rank.

Conceptually:

GPU 0 ─┐
GPU 1 ─┼── combine ──→ result on GPU 0
GPU 2 ─┼──────────────→ result on GPU 1
GPU 3 ─┘──────────────→ result on GPU 2
                      → result on GPU 3

Other collective operations include:

• AllGather
• ReduceScatter
• Broadcast
• AllToAll

Different parallelism implementations use different combinations of them.

For tensor-parallel inference, the useful principle is simple:

GPUs need to exchange intermediate tensors quickly enough that communication does not erase the benefit of splitting the compute.

Why GPU communication becomes part of inference performance

A single-GPU model performs all of its relevant operations locally.

A tensor-parallel model might do:

compute
↓
communicate
↓
compute
↓
communicate
↓
compute

through many model layers.

Adding GPUs reduces the amount of computation and model state assigned to each GPU.

It also introduces synchronization and communication.

That creates a tradeoff.

If each GPU saves a lot of compute while the communication path is fast, tensor parallelism can work very well.

If the model shard becomes small while communication remains expensive, adding another GPU can produce much less benefit.

This is why:

2 GPUs

does not imply:

2× inference performance

and:

8 GPUs

certainly does not imply:

8× performance

for one request.

Multi-GPU communication is something Hivenet can actually measure

This is where we have more useful evidence than a generic tensor-parallelism explainer.

Compute with Hivenet has tested NCCL AllReduce on a single host with eight RTX 5090 GPUs.

The measured AllReduce bus bandwidth at a 1GB message size was:

‍

‍

The 0.5% difference was within normal run-to-run variance, so we do not interpret it as the VM being faster. The useful result is that, for this communication benchmark on the tested 8-GPU host, the VM matched the bare-metal baseline. The result has also held across thousands of repeated benchmark runs.

You can read the full methodology in our GPU VM vs bare-metal benchmark.

That test does not prove that every tensor-parallel model will match bare-metal end-to-end latency.

It measures the communication path that workloads such as tensor-parallel inference rely on.

The distinction matters.

What the NCCL benchmark does and does not prove

The benchmark tells us that the tested Hivenet VM did not introduce a measurable AllReduce bandwidth penalty versus the bare-metal baseline on one 8 × RTX 5090 host.

It does not prove that:

• every model scales well to eight GPUs
• every tensor-parallel degree is efficient
• every message size behaves identically
• storage or CPU never becomes a bottleneck
• multi-node communication performs like single-host communication
• eight GPUs are better than four for your workload

Hivenet's test deliberately limits its claim to the communication pattern it actually measured.

That is also how you should evaluate tensor parallelism.

Benchmark the application.

Why doubling GPUs rarely doubles speed

Imagine one GPU takes:

100 units of time

to perform a workload.

If two GPUs split all computation perfectly in half, compute might fall toward:

50 units

But now add:

communication
synchronization
kernel-launch overhead
less efficient smaller matrix operations

Perhaps the real result becomes:

65 units

You still gained performance.

You did not gain 2×.

As TP degree rises, the amount of work per GPU becomes smaller while the need to coordinate remains.

Eventually, another GPU can cost more than it saves.

Where that point lies depends on the model, batch size, sequence length, precision, kernels, and hardware topology.

Latency and throughput can disagree

Suppose moving from TP=2 to TP=4 makes one request finish sooner.

That sounds like an obvious win.

But you are now using four GPUs for that request rather than two.

If the four-GPU server produces only a modest latency improvement, throughput per GPU may be worse.

These answer different questions:

Latency
How long does one request wait?

Total throughput
How much work does the server complete?

Throughput per GPU
How efficiently are you using the hardware you pay for?

A serving configuration can improve the first while making the third worse.

Choose the metric that belongs to the product.

Tensor parallelism versus pipeline parallelism

The distinction is worth making explicit:

‍

‍

With tensor parallelism:

Layer 1 → GPU 0 + GPU 1 + GPU 2 + GPU 3
Layer 2 → GPU 0 + GPU 1 + GPU 2 + GPU 3
Layer 3 → GPU 0 + GPU 1 + GPU 2 + GPU 3

With pipeline parallelism:

GPU 0 → layers 1–20
GPU 1 → layers 21–40
GPU 2 → layers 41–60
GPU 3 → layers 61–80

Current vLLM guidance recommends pipeline parallelism when tensor parallelism has already been used efficiently and the model still needs further distribution, or when layer-wise distribution suits the architecture better. It also notes that PP can be preferable when the model cannot be divided evenly across the available TP configuration or when the GPU interconnect makes TP communication expensive.

They can also be combined.

Tensor and pipeline parallelism can work together

Suppose you have eight GPUs.

You could configure:

tensor parallel size = 4
pipeline parallel size = 2

giving:

4 × 2 = 8 GPUs

Conceptually:

Pipeline stage 1
GPU 0 + GPU 1 + GPU 2 + GPU 3
       ↓
Pipeline stage 2
GPU 4 + GPU 5 + GPU 6 + GPU 7

Inside each stage, four GPUs use tensor parallelism.

Between stages, the model is divided by layers.

vLLM supports this combined configuration for models requiring larger distributed deployments.

The correct mixture depends on the hardware and model.

There is no requirement to use every parallelism technique just because the framework supports it.

Tensor parallelism versus data parallelism

Data parallelism solves a different problem.

Tensor parallelism:

one model
split across GPUs

Data parallelism:

complete model replica
on several GPU groups

Each data-parallel replica can serve different requests.

vLLM currently recommends data parallelism when the full model already fits within each replica and the main goal is increasing serving throughput rather than making the model itself fit.

For example, if a model fits on one GPU and you have four GPUs:

DP = 4

GPU 0 → full model → request set A
GPU 1 → full model → request set B
GPU 2 → full model → request set C
GPU 3 → full model → request set D

That can be much more efficient than TP=4 if the model does not need to be sharded.

Why make four GPUs coordinate on every request if each one can serve the model independently?

The better configuration may combine TP and data parallelism

Imagine a 70B model that requires two GPUs after quantization.

With eight GPUs, you could make:

4 replicas
×
2-way tensor parallelism

Each model replica spans two GPUs.

Four replicas process independent request batches.

Conceptually:

Replica A → GPU 0 + 1
Replica B → GPU 2 + 3
Replica C → GPU 4 + 5
Replica D → GPU 6 + 7

That can be more useful for a high-throughput API than one:

TP = 8

replica where all eight GPUs collaborate on every request.

vLLM supports combining TP and data parallelism in this way.

Again, benchmark it.

What about Mixture-of-Experts models?

MoE models create another parallelism option.

Instead of slicing every expert across every GPU, expert parallelism can place different experts on different devices.

Current vLLM supports a dedicated expert-parallel strategy for MoE models and distinguishes it from normal tensor parallelism.

That is particularly relevant to models such as DeepSeek-style MoEs where only some experts are active for each token.

We'll cover the hardware implications separately in our planned Mixture of Experts GPU memory guide.

Tensor parallelism still matters in MoE deployments.

It just stops being the only useful way to split the model.

Should you quantize or use tensor parallelism?

This is one of the most useful infrastructure decisions in the whole cluster.

Suppose a model at BF16 requires two GPUs but a good NVFP4 checkpoint fits on one.

You have two possible paths:

Path A

higher precision
+
tensor parallelism
+
more GPUs

Path B

lower precision
+
one GPU
+
no tensor-parallel communication

Neither is automatically better.

Our NVFP4 guide explains what the quantization path changes.

The tensor-parallel path preserves more precision but adds hardware, communication, synchronization, and cost.

When quantization is probably the better first move

I would test quantization first when:

• a validated quantized checkpoint exists
• it passes your quality evaluation
• it reduces the model enough to fit comfortably on one GPU
• you do not need enormous KV-cache capacity
• single-GPU latency and operational simplicity matter
• cost matters

Moving from two GPUs to one removes an entire distributed-computing problem.

That can be worth a lot.

If the quantized model behaves identically enough for your workload, adding a second GPU just to preserve unused precision may not be a good trade.

When tensor parallelism is the better choice

I would favor TP when:

• higher precision materially improves the task
• no good quantized checkpoint exists
• the model still does not fit after acceptable quantization
• you need more KV-cache capacity
• you need lower per-request latency and benchmarking shows TP helps
• the serving workload can justify the additional GPUs
• the communication path is good enough

There is also a third option:

quantization
+
tensor parallelism

A quantized 70B model might technically fit across two GPUs but run better across four because the extra cards leave more cache and concurrency headroom.

The techniques are complementary.

The smallest TP size that works is a good starting point

I would not start a deployment by asking:

How do we use all eight GPUs?

Start with:

What is the smallest TP degree that satisfies the workload?

If:

TP = 2

fits the model and provides enough cache, test it.

Then compare:

TP = 4

using exactly the same model and workload.

If four GPUs materially improve the metric you care about, keep them.

If they give a small gain for twice the hardware, TP=2 may be the better deployment.

Unused GPUs can support another replica.

What should you benchmark?

For inference, track at least:

Time to first token (TTFT)
How long until generation begins?

Inter-token latency
How quickly do subsequent tokens arrive?

Output throughput
How many tokens per second are produced?

Total request throughput
How many requests can the service sustain?

Concurrency
How does performance change when several users are active?

VRAM per GPU
Did TP create the cache headroom you expected?

GPU utilization
Are the GPUs doing useful compute or waiting?

Communication time
How much of the request is spent coordinating GPUs?

Those metrics tell you whether another GPU actually solves a problem.

How to test communication separately

NCCL's nccl-tests package includes benchmarks such as:

all_reduce_perf

which isolate collective communication rather than measuring the full LLM.

That is the benchmark Hivenet used for its VM-versus-bare-metal test.

A communication benchmark is useful because it separates:

GPU communication problem

from:

model or inference-engine problem

If NCCL performance is healthy but the model scales badly from TP=4 to TP=8, the bottleneck may be in the workload rather than the basic GPU communication path.

Check the GPU topology

The physical path between GPUs matters.

On a multi-GPU Linux machine, NVIDIA provides:

nvidia-smi topo -m

to inspect the topology the system exposes.

The broader principle is important even if you never read a PCIe topology table manually:

two GPUs being in the same machine does not guarantee that every pair communicates through an equally efficient path.

CPU sockets, PCIe root complexes, switches, and accelerator interconnects can affect communication.

Hivenet's multi-GPU benchmark explicitly tested whether the VM exposed the underlying single-host topology well enough for NCCL to perform as expected, which is why the result is more useful than a single-GPU compute benchmark for this particular question.

Single-node tensor parallelism is the easier case

When all tensor-parallel GPUs live inside one server, communication remains within that machine.

Once tensor parallelism crosses machine boundaries, the network becomes part of every distributed operation.

vLLM's parallelism guidance warns that efficient multi-node tensor parallelism needs fast inter-node communication and recommends high-speed networking for that reason.

This is why a common design is:

tensor parallelism
within a node

+

pipeline or data parallelism
across nodes

rather than assuming one enormous TP group should span the whole cluster.

For the Hivenet deployments in this content cluster, our focus is primarily single-host multi-GPU inference.

That is also the scope of our published NCCL measurement.

What does tensor parallelism cost on RTX 5090s?

Hivenet currently publishes RTX 5090 Compute from €0.75 per GPU-hour, with per-second billing.

At that rate, the simple GPU-time calculation is:

These numbers do not tell you cost per request.

‍

‍

If TP=4 serves twice as much useful work as TP=2 but costs twice as much, the economics may be roughly flat.

If TP=4 improves latency by 15% while doubling the GPU bill, the extra hardware is expensive latency.

Measure:

total GPU cost
÷
useful work completed

rather than choosing based on hourly price alone.

More aggregate VRAM can be useful even when scaling is poor

Suppose TP=4 does not deliver dramatically more tokens per second than TP=2.

It may still be the right configuration if it lets you support:

• a longer context window
• more concurrent sequences
• FP8 instead of FP4
• a larger model
• fewer OOM failures

Performance is not only decode speed.

Capacity can be the reason you added GPUs.

Be clear which problem the additional cards solve.

Common tensor-parallelism mistakes

“Two 32GB GPUs equal one 64GB GPU”

They give you 64GB of aggregate VRAM.

The framework still has to shard the model and coordinate operations across two separate devices.

“TP=4 should be four times faster”

The compute is divided, but communication and synchronization are added.

Scaling is workload-dependent.

“Use all available GPUs”

The economically useful TP size may be smaller than the number of GPUs you own.

Remaining GPUs can sometimes become additional model replicas.

“The model fits, so TP is unnecessary”

You may still use TP to create more KV-cache space or meet latency requirements.

“The model doesn't fit, so TP is the only answer”

Quantization, a larger-memory GPU, pipeline parallelism, or another model may be better.

“Aggregate VRAM is enough to size the deployment”

Model sharding, unsharded allocations, runtime memory, and KV cache all matter.

A practical decision tree

If the model fits comfortably on one GPU:

Start with TP = 1

Benchmark that first.

If the model barely fits but leaves too little KV cache:

compare quantization
vs
TP = 2

If the model does not fit on one GPU:

test the smallest TP size
that can hold the actual checkpoint
with useful memory headroom

If TP begins scaling poorly:

test pipeline parallelism
or another TP/PP combination

If you need more throughput, rather than one larger model replica:

add data-parallel replicas
instead of endlessly increasing TP

If the model is MoE:

evaluate expert parallelism too

This is the more useful framework than “large model = more GPUs.”

Tensor parallelism is a model-placement decision first

The term sounds like a performance optimization.

Often its first job is simpler:

make the model possible.

A 140GB model cannot live on a 32GB GPU.

Splitting it across enough GPUs creates a valid memory layout.

Once the model fits, you can start optimizing:

• precision
• TP degree
• cache
• batching
• latency
• throughput
• replica count

That order matters.

A system cannot optimize an inference configuration that cannot load.

And it is a communication problem second

Once you split the model, every GPU stops being independent.

They become participants in one distributed computation.

That means performance now depends on:

GPU compute
+
GPU memory
+
GPU-to-GPU communication
+
synchronization
+
serving-engine implementation

This is why multi-GPU inference belongs to systems engineering rather than GPU arithmetic.

Adding VRAM is easy.

Using it efficiently across several devices is the real problem.

Tensor parallelism FAQ

What is tensor parallelism?

Tensor parallelism splits tensors and model operations inside individual neural-network layers across several GPUs. Each GPU computes part of the layer and communicates with the others to produce the complete result. vLLM describes it as a common strategy for large-model inference within one node.

Why is tensor parallelism used for LLMs?

It lets models too large for one GPU run across several GPUs. It can also reduce model-memory pressure per GPU, leaving more VRAM available for KV cache and serving concurrency.

Does tensor parallelism combine GPU VRAM?

It allows one model to use aggregate memory across several GPUs through sharding, but it does not turn those GPUs into one physical shared-memory device. The serving framework must explicitly distribute the model and coordinate the computation.

What does tensor parallel size mean?

Tensor parallel size is the number of GPUs participating in one tensor-parallel model replica. --tensor-parallel-size 4 tells vLLM to distribute the supported model across four GPUs.

How do I enable tensor parallelism in vLLM?

For four GPUs:

vllm serve <model> \
 --tensor-parallel-size 4

vLLM also exposes tensor_parallel_size through its Python LLM class.

What is the difference between tensor parallelism and pipeline parallelism?

Tensor parallelism splits operations inside model layers across GPUs. Pipeline parallelism places different groups of layers on different GPUs and passes activations between those stages.

What is the difference between tensor parallelism and data parallelism?

Tensor parallelism divides one model replica across several GPUs. Data parallelism keeps separate complete model replicas and sends different request batches to each replica.

Can tensor parallelism and pipeline parallelism be combined?

Yes. vLLM supports configurations using both. For example, TP=4 and PP=2 use eight GPUs in total.

Can tensor parallelism and data parallelism be combined?

Yes. A model requiring two GPUs per replica could use TP=2 and then run several data-parallel replicas across a larger GPU pool. vLLM supports combined TP and DP configurations.

Does tensor parallelism make inference faster?

It can reduce per-request compute and latency, but scaling is not linear because GPUs must exchange and synchronize intermediate results. The actual gain depends on model architecture, workload, TP degree, and communication performance.

Why does tensor parallelism need fast GPU communication?

The GPUs repeatedly need to combine or exchange partial tensor results. Collective operations such as AllReduce, AllGather, and ReduceScatter are commonly used in distributed GPU workloads. NVIDIA's NCCL provides these communication primitives.

What is NCCL AllReduce?

AllReduce combines values across participating GPU ranks using an operation such as a sum and returns the reduced result to every rank.

Does Hivenet support multi-GPU tensor-parallel workloads?

Hivenet currently offers RTX 5090 configurations from one to eight GPUs per instance. Its published single-host NCCL AllReduce benchmark on eight RTX 5090s measured 19.34 GB/s inside the VM versus 19.25 GB/s on bare metal, a difference within run-to-run variance.

Is tensor parallelism better than quantization?

They solve different problems. Quantization reduces model precision and memory. Tensor parallelism distributes a model across GPUs. If a quantized model passes your quality requirements and fits comfortably on one GPU, it may be simpler and cheaper. The two techniques can also be used together.

What is the best tensor-parallel size?

There is no universal value. Start with the smallest TP size that fits the actual checkpoint with enough room for KV cache and runtime allocations, then benchmark larger configurations only when they solve a measurable latency, throughput, or capacity problem.

Do more GPUs always increase throughput?

No. Higher TP degrees reduce per-GPU compute but add more communication and synchronization. At some point, the extra GPU can provide too little performance or capacity benefit to justify its cost.

When should I use data parallelism instead?

If the complete model already fits within one GPU or one small TP group and your goal is serving more independent requests, replicating the model with data parallelism can be more efficient than increasing tensor-parallel size.

What parallelism should I use for Mixture-of-Experts models?

Tensor parallelism can still be used, but expert parallelism may distribute expert networks more naturally. Current vLLM supports dedicated expert-parallel execution for MoE models.

‍