Mixture of Experts explained: active parameters, GPU memory, and MoE inference

DeepSeek-R1 has 671 billion total parameters and activates about 37 billion parameters for each token.

So is it a 671B model or a 37B model?

For storage and much of the deployment problem, 671B matters.

For how much of the network participates in processing an individual token, 37B matters.

Both numbers describe the same model. They answer different questions. DeepSeek publishes both figures for R1 and V3, which share the underlying Mixture-of-Experts architecture.

That distinction is the central idea behind a Mixture of Experts, or MoE.

An MoE model contains many parameterized sub-networks called experts. A learned routing system decides which experts should process each token. Only a subset is activated, so the model can have far more total parameters than it uses for one token's forward pass.

This is sometimes summarized as “more parameters for the same compute.”

Useful shorthand, but incomplete.

Those inactive expert weights have not disappeared. They still need storage. If you want fast GPU-resident inference, they generally need to live somewhere across the GPU system.

That is why MoE changes the relationship between model capacity, compute, memory, and communication rather than simply giving you a giant model at the cost of a small one.

What is a Mixture of Experts?

A conventional dense transformer sends every token through the same feed-forward network inside each transformer block.

Conceptually:

Token
 ↓
Attention
 ↓
Feed-forward network
 ↓
Next layer

A sparse Mixture-of-Experts transformer replaces some of those dense feed-forward operations with a collection of expert networks.

Conceptually:

                 ┌→ Expert 1 ─┐
                 │            │
Token → Router ───┼→ Expert 2 ─┼→ combine → next layer
                 │            │
                 ├→ Expert 3 ─┤
                 │            │
                 └→ Expert 4 ─┘

The router scores the experts for the current token.

It might select:

Expert 2
+
Expert 4

while another token in the same sentence might be sent to:

Expert 1
+
Expert 3

The experts belong to one model. They are not independent chatbots voting on an answer.

They are specialized parameter blocks inside the network.

Sparse expert architectures are built around this basic principle: different inputs activate different subsets of the available model parameters. The Switch Transformer work showed how sparse routing can scale model parameter counts dramatically without increasing per-example computation in the same proportion.

The router decides which experts run

The router, sometimes called a gating network, sits in front of the experts.

For each token, it calculates scores representing which experts should receive that token.

Imagine eight experts:

E1  E2  E3  E4  E5  E6  E7  E8

The router might produce:

0.02
0.05
0.61
0.03
0.18
0.04
0.05
0.02

If the model uses top-2 routing, the token goes primarily to:

E3
+
E5

The next token can produce completely different routing scores.

That is important.

Experts are not normally assigned permanently to obvious categories such as:

Expert 1 = French
Expert 2 = Python
Expert 3 = mathematics

Some specialization may emerge, but the router learns its own internal organization during training.

Mixtral gives us a clean example

Mistral's Mixtral 8x7B is one of the easiest MoE architectures to understand.

Each MoE layer contains:

8 experts

and the router sends each token through:

2 experts

at that layer.

The Mixtral paper reports approximately:

47B total parameters
13B active parameters per token

even though the model was commonly described as “8x7B.”

So one token does not use all eight experts.

It uses two.

At the next layer, the router decides again.

And another token can follow a different expert path entirely.

This is sparse activation.

Active parameters and total parameters measure different things

Consider:

Total parameters = 47B
Active parameters = 13B

The first tells us how many parameters exist across the model.

The second tells us approximately how many participate in the computation for a token.

That gives two different infrastructure questions.

Total parameters help determine storage

If the model contains 47 billion parameters, those weights exist somewhere.

A simple BF16 storage estimate starts around:

47B × 2 bytes
≈ 94GB

before checkpoint metadata and other model components.

Active parameters help describe computation

If about 13B parameters participate for a token, its per-token compute is much closer to that active path than to running a dense 47B network.

This is the central MoE trade.

large parameter capacity

without

activating every parameter
for every token

Why “active parameters” do not tell you the VRAM requirement

This is where MoE hardware advice often goes wrong.

Take DeepSeek-R1:

671B total parameters
37B active parameters

Someone sees 37B and reasons:

37B × 0.5 bytes
≈ 18.5GB at 4-bit

and concludes:

DeepSeek-R1 should fit on a 24GB GPU.

It doesn't work that way.

The other experts still contain weights.

DeepSeek-R1 uses the DeepSeek-V3 architecture, and DeepSeek explicitly reports 671B total parameters with 37B activated for each token.

A crude storage calculation for the complete 671B model is:

These are only weights-only approximations.

‍

‍

But they immediately show why:

37B active

does not mean:

hardware requirements of a 37B dense model

Our DeepSeek-R1 model-size and memory guide goes through that deployment problem in more detail.

So what does the 37B number tell us?

It tells us how much of the network participates in processing a token.

Very roughly, a dense 671B model would involve computations through a vastly larger set of weights for each token.

An MoE model can keep enormous parameter capacity available while choosing a much smaller route for the current token.

This is why MoE architectures are attractive when scaling models.

The Switch Transformer work describes sparse activation precisely in these terms: models can grow to extremely large parameter counts while keeping per-example computational cost much more controlled.

DeepSeek follows the same broader idea at much larger modern scale.

MoE does not make 671B compute like a generic dense 37B model

There is another oversimplification to avoid.

If DeepSeek-R1 activates 37B parameters, that does not mean:

DeepSeek-R1 inference cost
=
any random dense 37B model

Architecture still matters.

An MoE model includes:

• attention layers
• routers
• shared parameters
• expert layers
• communication between expert placements
• model-specific mechanisms such as DeepSeek's MLA
• different memory-access patterns

“37B active” is useful for understanding sparsity.

It is not a universal performance conversion factor.

Measure the actual model.

Dense model versus MoE

A useful high-level comparison is:

‍

‍

MoE buys parameter capacity without requiring all experts to execute simultaneously.

The cost is a more complicated model and serving system.

Where do the experts usually live?

In transformer MoEs, the expert networks commonly replace or augment the feed-forward/MLP portion of certain transformer blocks.

So a simplified dense block might be:

Attention
  ↓
Dense MLP

while an MoE block might be:

Attention
  ↓
Router
  ↓
Expert MLPs

The attention mechanism itself is usually not transformed into a large set of independently routed experts in the same way.

That matters because not every parameter in an MoE model is sparsely activated.

There are shared parts of the network that every token still uses.

DeepSeek also uses shared experts

DeepSeekMoE adds another useful idea.

Alongside routed experts, it isolates shared experts intended to capture common knowledge, while the routed experts can become more specialized. DeepSeek's published MoE architecture combines fine-grained expert segmentation with shared experts to reduce redundancy between routed specialists.

Conceptually:

Token
 │
 ├────────→ Shared expert ─────────┐
 │                                │
 ↓                                │
Router                             │
 │                                │
 ├→ Routed expert A ──────────────┤
 └→ Routed expert F ──────────────┤
                                  ↓
                             combine output

The shared expert is available broadly.

The router still selects a subset of routed experts.

That means DeepSeek's active-parameter count includes more than simply:

number of selected experts
×
expert size

The model has shared computation too.

Expert specialization is the reason to have experts at all

Suppose every expert learned exactly the same thing.

Then having 64 experts would mostly duplicate parameters.

The value of routing comes from experts learning sufficiently different functions so that different tokens benefit from different expert paths.

The original DeepSeekMoE work focuses heavily on this problem and explicitly tries to encourage more specialized experts while reducing redundant knowledge across them.

This also creates a training challenge.

You want specialization.

You do not want:

95% of tokens → Expert 7

while most other experts sit unused.

That is the load-balancing problem.

Expert load balancing matters

Routers do not necessarily distribute tokens evenly by themselves.

Some experts can become much more popular than others.

If one GPU contains the hot experts while another GPU contains rarely selected experts, the first becomes a bottleneck.

You could have:

GPU 0 → 95% busy
GPU 1 → 90% busy
GPU 2 → 25% busy
GPU 3 → 10% busy

even though the total expert weight allocation looks perfectly balanced on paper.

This is one reason MoE serving introduces a systems problem beyond weight storage.

The question becomes:

Are the experts distributed according to the traffic the router actually produces?

Current vLLM documentation notes that real token distributions can be highly skewed even when MoE models are trained to encourage balanced expert use. Its Expert Parallel Load Balancer can redistribute expert mappings across ranks based on observed load.

What is expert parallelism?

Dense-model serving commonly uses tensor parallelism to split large tensors inside model layers across GPUs.

MoE models give us another option:

expert parallelism.

Instead of slicing every expert across every GPU, expert parallelism places different experts on different GPUs.

Conceptually:

GPU 0
Expert 1
Expert 2

GPU 1
Expert 3
Expert 4

GPU 2
Expert 5
Expert 6

GPU 3
Expert 7
Expert 8

A token routed to:

Expert 2 + Expert 7

therefore needs work on:

GPU 0 + GPU 3

vLLM's current expert-parallel deployment explicitly distributes MoE experts across different GPUs while handling other layer types according to the tensor/data-parallel configuration.

Expert parallelism saves memory by distributing expert weights

Imagine:

64 experts

and:

8 GPUs

A simple expert-parallel placement might keep:

8 experts per GPU

rather than every GPU holding all 64.

That makes enormous expert collections much easier to store.

The cost comes when a token needs an expert that lives on another GPU.

Now its hidden state has to travel to the GPU holding that expert.

After expert computation, the result has to be returned or redistributed.

This is why MoE inference becomes communication-heavy.

Expert parallelism uses All-to-All communication

Tensor parallelism often relies heavily on collective patterns such as AllReduce.

Expert parallelism has a different natural communication pattern:

All-to-All.

Tokens initially exist on the GPUs processing their current batch.

The router determines which expert should process each token.

Those experts may live on different GPUs.

So tokens need to be dispatched:

GPU 0 tokens ─┬→ GPU 1 experts
             ├→ GPU 2 experts
             └→ GPU 3 experts

GPU 1 tokens ─┬→ GPU 0 experts
             ├→ GPU 2 experts
             └→ GPU 3 experts

Then expert outputs need to return to the appropriate sequences.

Current vLLM expert-parallel implementations provide several All-to-All backends, including general-purpose and specialized high-throughput or low-latency paths.

So once again, the model is not merely a memory problem.

GPU communication becomes part of inference.

Tensor parallelism and expert parallelism solve different problems

Compare them:

‍

‍

A large MoE deployment can use both.

Attention layers might use tensor parallelism while the experts use expert parallelism.

vLLM currently supports exactly this kind of mixed arrangement.

Why expert parallelism can outperform tensor-sharding every expert

Suppose each expert is large.

You could tensor-shard every expert across all GPUs.

Then every time a token uses an expert, many GPUs may participate in the operation.

Alternatively, expert parallelism can keep individual experts local to particular devices.

That can improve locality and reduce how broadly each expert computation has to be distributed.

vLLM describes expert parallelism as a way to increase locality and efficiency for MoE deployments.

But this pushes pressure onto token dispatch.

A poor expert layout can create expensive or imbalanced All-to-All communication.

Nothing about MoE eliminates distributed-systems tradeoffs.

It changes which tradeoffs matter.

DeepSeek-R1 is an MoE, but its distill models are not

This distinction is particularly important because the names are confusing.

Full:

DeepSeek-R1

is based on the DeepSeek-V3 MoE architecture:

671B total
37B active

But:

DeepSeek-R1-Distill-Qwen-14B
DeepSeek-R1-Distill-Qwen-32B
DeepSeek-R1-Distill-Llama-70B

are smaller dense models fine-tuned using reasoning data generated by R1.

DeepSeek explicitly says its R1 distill models are based on Qwen and Llama dense checkpoints.

So:

R1-Distill-Qwen-32B

does not contain a small copy of the R1 Mixture-of-Experts architecture.

It is a 32B Qwen-based dense model trained from R1-generated examples.

This is why our DeepSeek-R1 hardware guide treats the full model and the distills as completely different infrastructure problems.

Mixtral shows why the model name can also be misleading

Mixtral 8x7B sounds at first like:

8 × 7B
=
56B

but its published parameter accounting is about:

47B total
13B active

because not every part of the architecture is simply eight independent copies of a 7B model.

The attention layers and other shared structures are not replicated as eight entirely separate networks.

Mistral's paper explains that each MoE layer contains eight feed-forward experts and routes every token through two.

This is why parameter counts should come from the actual model documentation rather than multiplying numbers in the product name.

Wan 2.2 shows MoE outside language models

MoE is no longer only an LLM architecture.

Wan 2.2 applies an MoE design to video diffusion.

Its A14B models use two experts:

• a high-noise expert for the earlier denoising stage
• a low-noise expert for later refinement

Each expert has about 14B parameters.

Together the model contains roughly:

27B total parameters

while:

~14B

are active during a denoising step.

That routing mechanism is different from a language-model router selecting experts separately for arbitrary tokens.

Wan routes according to where the generation process is in the noise schedule.

The underlying idea is still recognizable:

increase total model capacity without using every expert for every operation.

Our Wan 2.2 ComfyUI and GPU guide covers that workload in detail.

Does MoE save GPU memory?

This deserves a precise answer.

Sparse activation does not automatically reduce the memory required to store the full model.

If you keep every expert resident on one GPU, all of their weights still consume VRAM even though only a subset is active for a token.

MoE can reduce memory pressure per GPU when experts are distributed across several devices using expert parallelism.

You can also reduce model storage through:

• quantization
• CPU offloading
• expert offloading
• specialized placement
• distributed serving

But these are deployment techniques.

The basic sparse-routing architecture primarily reduces the amount of expert compute activated per token relative to activating every expert.

Does an inactive expert consume compute?

For that token, not in the same way as a selected expert.

That is the point of sparse activation.

If a router selects:

2 of 8 experts

the other six expert MLPs do not all need to perform their full forward computation for that token.

But their parameters still exist.

And on a GPU-resident implementation, those weights can still occupy VRAM.

So:

inactive

means:

not selected for this token's expert computation

rather than:

does not exist in memory

Does MoE reduce FLOPs?

Relative to a dense network with the same total parameter count, often dramatically.

That is one of its main purposes.

Mixtral exposes 47B parameters while activating about 13B per token.

DeepSeek exposes 671B total parameters while activating 37B per token.

The goal is to decouple:

how much parameter capacity exists

from:

how much computation each token uses

The result is sparse computation.

That does not imply the same latency as a dense model matching the active parameter count, because routing, memory access, and communication remain different.

Does MoE make inference faster?

Compared with what?

This question needs a baseline.

A 671B dense model would require enormous computation for every token.

A 671B-total MoE that activates 37B is far cheaper than that hypothetical dense 671B path.

But a distributed 671B MoE is not necessarily faster than a well-optimized dense 32B model.

MoE inference can incur:

• routing overhead
• irregular expert loads
• All-to-All communication
• expert synchronization
• large total weight storage
• less predictable memory access

Sparse compute is useful.

Distributed sparse compute is still systems engineering.

Why model quality cannot be predicted from active parameters

Suppose we compare:

MoE model
37B active
671B total

with:

dense model
40B total
40B active

It would be tempting to say:

They should have roughly the same capability because they use roughly the same parameters per token.

That does not follow.

The MoE can draw on different parameter subsets for different tokens.

Across a document or conversation, many more of its total parameters may participate at different times.

Training data, architecture, routing quality, optimization, context design, and post-training also matter.

Parameter count was never a complete capability metric for dense models.

MoE makes it even less complete.

Can different tokens use different experts?

Yes.

That is fundamental to token-routed language MoEs such as Mixtral.

Consider:

"The Python function returns a list."

At one layer, the router might produce:

"The"      → Experts 2, 4
"Python"   → Experts 1, 7
"function" → Experts 3, 7
"returns"  → Experts 3, 5
"a"        → Experts 2, 4
"list"     → Experts 1, 6

Those are only illustrative IDs.

The important point is that routing can vary by token and by layer.

Mixtral explicitly routes each token independently to two experts in every MoE layer.

This creates enormous combinations of possible expert paths without requiring every expert to run every time.

Experts are not necessarily human-interpretable specialists

It would be convenient if you could open a model and discover:

Expert 12 = medicine
Expert 17 = JavaScript
Expert 24 = Greek

Real learned specialization is usually messier.

An expert might respond to patterns that do not correspond neatly to one human category.

Specialization can also vary across layers.

Earlier experts may respond to different features than later ones.

The model does not need experts to come with understandable job titles.

It needs the routing and expert representations to improve prediction.

What happens if every token chooses the same expert?

That creates a serious problem.

One expert becomes overloaded.

Others receive too little training or inference work.

The model then loses much of the benefit of having many experts.

Training MoE systems therefore use routing and balancing mechanisms intended to stop severe expert collapse.

The Switch Transformer and later MoE work discuss routing capacity and expert balancing as central challenges, while current serving systems also need to cope with expert skew in real traffic.

This is why the router is not a trivial classifier bolted onto the model.

It is part of the architecture's success or failure.

Load balancing also becomes a serving problem

Training may produce broadly balanced expert utilization over a huge corpus.

Your production traffic might not resemble that corpus.

Imagine a deployment used almost entirely for code.

Its prompts may consistently trigger a subset of experts more frequently.

Now one expert-parallel rank gets far more work than another.

The slowest overloaded rank can limit the whole distributed forward pass.

Current vLLM provides an Expert Parallel Load Balancer that tracks expert load and can adjust expert mappings across ranks to mitigate this skew.

That is a good example of an inference problem that dense models simply do not have in the same form.

Expert replication can help hot experts

Another strategy is to replicate frequently used experts.

Instead of one copy of:

Expert 17

serving every token routed there, several devices can hold copies.

That trades additional memory for more compute capacity and better load distribution.

The same principle appears repeatedly in serving design:

memory
↔
throughput

Use extra memory to duplicate hot capacity.

Or save memory by keeping fewer copies and accept more contention.

The correct choice depends on the workload.

MoE makes communication topology increasingly important

With tensor parallelism, GPUs exchange pieces of tensor operations.

With expert parallelism, tokens may move toward whichever device owns their selected experts.

In both cases, a multi-GPU server's communication path matters.

For single-host Hivenet RTX 5090 configurations, we have measured NCCL AllReduce on an 8-GPU host and found VM bus bandwidth within normal variance of the bare-metal baseline. That result does not directly benchmark MoE All-to-All traffic, but it does establish that the virtualized multi-GPU environment can expose a high-performance GPU communication path for the tested collective workload.

The methodology and result are in our GPU VM vs bare-metal benchmark.

For an actual MoE deployment, benchmark the exact All-to-All and model workload rather than treating an AllReduce result as interchangeable.

MoE and tensor parallelism can be combined

Suppose a model has:

Attention layers
+
MoE expert layers

You can use:

tensor parallelism for the attention and other large dense tensors,

and:

expert parallelism for the expert layers.

Current vLLM supports this mixed design. With expert parallelism enabled, expert layers are distributed across the EP group while attention behavior depends on the TP and DP configuration.

This gives the serving engine different tools for different parts of the same model.

That is often more sensible than pretending every layer should be parallelized identically.

Expert parallelism and data parallelism can also work together

Data parallelism normally means:

replicate model
→ process independent request groups

For MoE deployments, things become more interesting.

vLLM can combine data parallelism with expert parallelism so expert weights are distributed across ranks rather than replicated independently within every data-parallel copy.

The result can improve expert locality and memory efficiency.

It also means supposedly “data-parallel” ranks may need synchronized participation in expert-layer execution.

MoE makes even familiar parallelism terminology more nuanced.

Quantization still matters for MoE

Sparse activation reduces compute.

It does not shrink stored expert weights.

Quantization can.

For an MoE containing hundreds of billions of total parameters, moving suitable weights from:

BF16

to:

FP8

or lower-precision formats can remove hundreds of gigabytes from the storage requirement.

That makes the NVFP4 and Blackwell inference guide relevant to MoE deployment too.

The combination can be powerful:

MoE
→ activate fewer experts

Quantization
→ make expert weights smaller

Expert parallelism
→ distribute experts across GPUs

Each technique attacks a different part of the infrastructure problem.

KV cache still exists in an MoE LLM

MoE does not remove attention.

So language-model MoEs still need attention state during autoregressive inference.

Their KV cache can still grow with:

• context
• concurrency
• architecture
• cache precision

This creates an important competition for VRAM.

A GPU may hold:

local expert weights
+
attention/shared weights
+
KV cache
+
runtime allocations

Distributing experts across more GPUs may therefore help both with fitting the model and with creating more room for cache.

Once again, active parameter count alone tells you very little about the total serving-memory problem.

PagedAttention helps the cache, not the expert weights

Our PagedAttention and continuous batching guide explains how vLLM can manage active KV-cache memory efficiently as requests grow and finish.

That does not page arbitrary hundreds of gigabytes of expert weights in and out for free.

These are separate memory layers:

Model parameters
→ experts + shared model weights

Inference state
→ KV cache + temporary state

MoE and PagedAttention solve different problems.

A good serving stack has to deal with both.

How should you estimate MoE memory?

Start with the total model weights, not active parameters.

If the full model checkpoint is available, use its actual size.

If all you have is parameter count, use a rough planning estimate:

total parameters
×
bytes per stored parameter

Then account for:

• quantization metadata
• higher-precision layers
• embeddings
• attention/shared model weights
• runtime buffers
• KV cache
• expert placement
• replication

If experts are distributed across GPUs, calculate how much expert state each GPU stores.

Then add the shared or sharded parts of the model.

This is more work than:

active params × 0.5

It is also much closer to reality.

A DeepSeek-R1 memory thought experiment

Start with:

671B total parameters
37B active parameters

If every parameter were hypothetically 4-bit:

671B × 0.5 bytes
≈ 335.5GB

Eight RTX 5090s provide:

8 × 32GB
=
256GB aggregate VRAM

So even the idealized raw 4-bit weight storage exceeds the entire eight-GPU pool.

This is why full R1 does not suddenly become a normal eight-5090 workload merely because only 37B parameters are activated for a token.

Expert sparsity reduces compute.

It does not erase the full expert parameter set.

Specialized quantization, offloading, and distributed deployments can change the implementation. The basic arithmetic explains why the starting point remains difficult.

An MoE can be compute-light relative to its storage

This is perhaps the best mental model.

MoE creates an unusual ratio:

a lot of stored parameters
↓
a smaller subset of active compute

Dense models keep those two quantities more tightly connected.

This makes MoEs attractive when compute is the limiting factor and you can accommodate the parameter storage and routing system.

It can be awkward when memory capacity or GPU communication is the primary constraint.

That is why MoE is not automatically a cheaper architecture in every deployment.

When does MoE make sense?

From a model-design perspective, MoE is compelling when you want:

• very large total model capacity
• sparse per-token computation
• experts that can specialize
• better scaling without proportionally increasing FLOPs

From an infrastructure perspective, it makes sense when the serving system can handle:

• large total weight storage
• routing
• expert placement
• All-to-All communication
• load imbalance
• expert parallelism
• cache and shared layers

The architecture and infrastructure have to fit each other.

When can a dense model be easier?

A dense model is operationally simpler.

Every token follows essentially the same parameter path.

There is no expert routing.

There are no hot experts.

You do not need expert placement or token All-to-All dispatch.

A strong dense 27B model that fits on one GPU can therefore be easier to operate than a much larger MoE even if the MoE's active parameter count sounds similar.

Our Qwen3.6-27B cloud GPU guide is a good example of the simpler deployment class.

Model architecture is part of operations.

Don't choose an MoE because the total parameter count is larger

A model with:

300B total
30B active

is not automatically superior to a:

70B dense

model.

And the reverse is equally unsupported.

Evaluate the model on the task.

Then evaluate the infrastructure needed to serve it.

The useful questions are:

1. Does the model produce better results?
2. How many parameters are actually stored?
3. How many are activated?
4. How much VRAM does the real checkpoint need?
5. How is it quantized?
6. How are experts distributed?
7. What communication does inference require?
8. What concurrency and context do we need?
9. What does useful throughput cost?

Parameter count belongs inside that decision.

It should not replace it.

The useful way to think about Mixture of Experts

A Mixture-of-Experts model gives the network more possible parameters than it needs to activate for one token.

The router chooses a path.

That gives MoE its main advantage:

large total capacity
+
sparse computation

The same design creates its main infrastructure challenge:

all those expert weights
still need somewhere to live

Once several GPUs are involved, another requirement appears:

tokens must reach
the experts they selected

So MoE inference becomes a balance between:

capacity
compute
memory
routing
communication
load balance

That is why “671B total, 37B active” is not contradictory.

It is a compact description of the architecture.

The 671B tells you how much parameter capacity exists.

The 37B tells you how much of that capacity participates in a token's computation.

You need both numbers to understand the model.

Mixture of Experts FAQ

What is a Mixture-of-Experts model?

A Mixture-of-Experts model contains multiple expert networks and a routing mechanism that selects a subset of those experts for each input or token. Sparse MoE models therefore activate only part of their total parameter set for each forward pass.

What does MoE stand for?

MoE stands for Mixture of Experts.

What is an expert in an MoE?

An expert is a parameterized sub-network inside the model, commonly a feed-forward/MLP block in transformer MoEs. The router chooses which experts process a token.

What is the router in an MoE?

The router is a learned gating mechanism that scores available experts and decides which subset should process each token.

What are active parameters?

Active parameters are the parameters participating in the computation for a particular token or forward path. They can be far fewer than the model's total parameter count in a sparse MoE.

What are total parameters?

Total parameters count all the learned parameters contained in the model, including experts that may not be selected for the current token.

Does an MoE need memory for inactive experts?

Yes. If the expert weights are resident on accelerators, inactive experts still occupy memory even when they are not selected for a specific token. They can be distributed or offloaded, but sparse activation itself does not erase their storage requirement.

Why does DeepSeek-R1 have 671B parameters but only 37B active?

DeepSeek-R1 uses the DeepSeek-V3 MoE architecture. It contains 671B total parameters, while the routing architecture activates about 37B parameters for each token.

Is DeepSeek-R1 effectively a 37B model?

No. The 37B figure describes activated parameters, not total stored model size. The complete model contains 671B parameters.

How much VRAM does a 671B MoE need?

It depends on precision, quantization, placement, and offloading. As a raw planning estimate, 671B parameters correspond to roughly 1.34TB at 16-bit, 671GB at 8-bit, or 335.5GB at 4-bit before other overhead.

Does MoE reduce compute?

Sparse MoE can substantially reduce computation relative to a dense model with the same total parameter count because only selected experts are activated for each input. This is one of the primary motivations for sparse expert architectures.

Does MoE reduce GPU memory?

Not automatically. Sparse activation reduces active computation. The full expert weights still need storage unless they are distributed, quantized, or offloaded.

What is expert parallelism?

Expert parallelism distributes different MoE experts across different GPUs. Tokens are routed between devices according to which experts they need. Current vLLM supports expert-parallel MoE deployment.

How is expert parallelism different from tensor parallelism?

Tensor parallelism splits large tensor operations across GPUs. Expert parallelism places different experts on different GPUs and routes tokens to them. MoE serving can use both techniques.

What communication does expert parallelism use?

Expert parallelism commonly requires All-to-All-style communication so token states can be dispatched to the GPUs holding selected experts and results can be returned. Current vLLM exposes multiple All-to-All backends for expert-parallel serving.

What is expert load balancing?

Expert load balancing tries to prevent a small number of experts or expert-hosting GPUs from receiving a disproportionate amount of work. vLLM currently offers an Expert Parallel Load Balancer that can adjust expert mappings based on observed load.

Is Mixtral 8x7B a 56B model?

Its published model accounting is approximately 47B total parameters with about 13B active per token. Each MoE layer contains eight experts and routes each token through two.

Are DeepSeek-R1 distill models MoE models?

No. The official R1 distills are dense Qwen and Llama-based models fine-tuned on data generated by DeepSeek-R1.

Is Mixture of Experts only used for LLMs?

No. Wan 2.2 uses an MoE architecture in its A14B video diffusion models, with separate high-noise and low-noise experts activated at different stages of denoising.

Does every token use the same experts?

Not necessarily. In token-routed language MoEs such as Mixtral, the router can choose different experts for different tokens and at different layers.

Are experts specialized in obvious topics?

Not necessarily. Experts can develop specialization, but their learned roles do not have to map cleanly onto human categories such as programming, medicine, or individual languages.

Is MoE faster than a dense model?

MoE can use much less compute than a dense model with the same total number of parameters, but actual inference speed depends on routing, expert placement, communication, memory access, hardware, and serving software.

Should I choose a model based on active parameters?

No. Active parameters help describe inference compute, but model quality and hardware requirements also depend on total parameters, architecture, training, precision, routing, context, and serving configuration.

‍