What is NVFP4? 4-bit AI inference on NVIDIA Blackwell explained

NVFP4 is a 4-bit floating-point format designed by NVIDIA for Blackwell GPUs.

Its purpose is straightforward: make large AI models smaller and cheaper to run without throwing away so much numerical information that the model stops being useful.

But calling NVFP4 simply “4-bit quantization” hides most of what makes it interesting.

The underlying number uses only four bits. NVIDIA then combines those values with fine-grained scaling, using an FP8 scale for small groups of values and a higher-precision global scale for the tensor. That lets the 4-bit values adapt to the numerical range of different parts of a model rather than forcing one coarse scale across a large tensor.

On Blackwell hardware, those values can also be processed by Tensor Cores with native FP4 support. The RTX 5090, for example, is a Blackwell GPU with fifth-generation Tensor Cores and 32GB of GDDR7 VRAM. NVIDIA's RTX Blackwell architecture adds hardware support for FP4 Tensor Core operations.

That combination matters.

A 27B model that is too large for one 32GB GPU at BF16 can become a practical single-GPU workload after careful NVFP4 quantization.

The important word is careful.

Good low-precision inference is not about turning every tensor into four bits and hoping for the best.

What does NVFP4 mean?

NVFP4 uses an E2M1 floating-point representation.

That means each four-bit value contains:

1 sign bit
2 exponent bits
1 mantissa bit

The raw E2M1 number can represent magnitudes including:

0
0.5
1
1.5
2
3
4
6

and their negative equivalents.

On its own, that is a very small numerical vocabulary.

Modern neural-network tensors contain values with much more varied distributions.

NVFP4 deals with that through scaling.

NVIDIA defines an NVFP4 value conceptually as:

high-precision value
=
4-bit E2M1 value
×
local FP8 scale
×
global FP32 scale

The local scale is stored using FP8 E4M3 and is shared across a small group of values. A second FP32 scale operates across the tensor.

So NVFP4 is better understood as:

4-bit values + a structured scaling system

rather than simply:

make every number four bits long.

Why does scaling matter so much?

Imagine these values:

0.02
0.04
0.07
0.11

and elsewhere in the same tensor:

2.1
3.4
4.8
5.7

A single low-precision scale that has to cover both groups may represent one region poorly.

The small values could collapse toward the same quantization levels, while the large values dictate the useful numerical range.

Fine-grained scaling lets separate groups adapt to their own local ranges.

NVFP4 normally uses a scale for every 16-element block, together with the global tensor scale. NVIDIA says this smaller block size reduces quantization error compared with formats that use larger scaling blocks.

For weights, NVIDIA's current Transformer Engine implementation can use a two-dimensional variant where 16 × 16 weight blocks share scaling information. Activations and gradients use the one-dimensional 16-element form.

You do not need to memorize that implementation detail to use an NVFP4 model.

The useful idea is:

the smaller the region sharing a scale, the better the quantizer can adapt to local differences in the tensor.

That precision costs some metadata.

It also helps explain why real NVFP4 models preserve substantially more useful information than the phrase “four bits” might suggest.

NVFP4 is not the same as plain FP4

FP4 describes the underlying four-bit floating-point representation.

NVFP4 describes a more complete quantization format and scaling strategy.

NVIDIA compares three Blackwell FP4 approaches:

All three may store four-bit data.

‍

‍

They do not necessarily preserve information equally well.

NVFP4's smaller blocks and higher-precision scaling are intended to reduce quantization error while still letting Blackwell perform low-precision matrix operations efficiently.

That is why asking:

Is this model FP4?

is less useful than asking:

Which FP4 format and quantization recipe does this checkpoint use?

NVFP4 versus FP8 versus BF16

The basic precision hierarchy looks like this:

‍

‍

The raw bit count is easy.

Real model sizes are not.

NVFP4 needs scaling data. Some layers may remain at FP8 or BF16. Embeddings, normalization layers, output heads, multimodal components, and other parts of a model may not use the same precision.

NVIDIA therefore reports about 3.5× lower model memory than FP16 and about 1.8× lower than FP8 in its published NVFP4 analysis rather than claiming the perfect theoretical 4× and 2× reductions.

This distinction matters when sizing GPUs.

Do not calculate:

54GB BF16 checkpoint
÷ 4
=
13.5GB

and conclude that the final deployed NVFP4 model will occupy exactly 13.5GB.

That number only describes idealized four-bit storage of the parameters.

A real checkpoint is a mixed collection of quantized values, scales, metadata, and components left at higher precision.

FP8 or NVFP4?

FP8 gives each main value twice as many bits as NVFP4.

That gives FP8 considerably more numerical resolution and range before scaling is considered.

NVFP4 trades some of that representation capacity for:

• lower model memory
• lower memory-bandwidth pressure
• more room for larger models
• more room for KV cache
• potentially higher low-precision Tensor Core throughput on Blackwell

So the choice is not:

accurate FP8 or inaccurate FP4.

It is:

Does this particular model retain the quality we need when its appropriate layers are moved to NVFP4?

Sometimes yes.

Sometimes no.

That is an evaluation problem.

What does W4A4 mean?

You will often see an NVFP4 model described as:

W4A4

The notation means:

W4 = 4-bit weights
A4 = 4-bit activations

Weights are the learned parameters stored in the model.

Activations are intermediate values produced while the model processes a request.

A weight-only 4-bit model might instead resemble:

W4A16

where:

weights = 4-bit
activations = 16-bit

W4A4 can reduce both model storage and the precision of the matrix operations themselves, which is particularly interesting on hardware with native 4-bit Tensor Core execution.

NVIDIA's Qwen3.6-27B NVFP4 checkpoint, for example, quantizes the weights and activations of linear operators to NVFP4 and serves the resulting model through vLLM.

This is why the format is more than a disk-compression trick.

The inference math itself can use the lower precision.

Weight quantization and activation quantization solve related problems

Reducing weight precision helps with:

• checkpoint size
• GPU memory
• memory bandwidth while reading weights

Reducing activation precision can also help make the actual compute path cheaper and better aligned with low-precision Tensor Cores.

That is why an NVFP4 W4A4 model can be different from a traditional four-bit checkpoint where weights are compressed but immediately combined with higher-precision activations.

Again, software support matters.

A checkpoint claiming four-bit weights does not automatically mean your runtime will perform native four-bit matrix operations.

Why Blackwell matters

NVFP4 was introduced with NVIDIA Blackwell.

Blackwell Tensor Cores add native FP4 support. On RTX Blackwell, NVIDIA also added FP6 and a newer FP8 Transformer Engine alongside the existing BF16, FP16, TF32, and INT8 paths.

That hardware support is central to the format.

A model can be stored in some four-bit representation on older hardware, but that does not mean the GPU can execute NVFP4 matrix operations natively.

For native NVFP4 acceleration, think Blackwell or newer.

Examples include Blackwell GPUs such as:

• GeForce RTX 5090
• RTX PRO 6000 Blackwell
• B200-class data-center GPUs

Exact runtime and kernel support still depends on the software stack.

The GPU architecture is necessary.

It is not the only dependency.

Why the RTX 5090 is interesting for NVFP4

The desktop RTX 5090 has:

32GB GDDR7 VRAM
1,792 GB/s memory bandwidth
Blackwell architecture
5th-generation Tensor Cores
FP4 support

according to NVIDIA's architecture documentation.

Thirty-two gigabytes is a useful amount of memory, but it is not enough for every modern model at BF16.

A 27B model needs approximately:

27B × 2 bytes
≈
54GB

for BF16 weights alone.

That model is outside a single RTX 5090 before inference begins.

Move suitable operations to NVFP4 and the calculation changes enough that a 27B-class model can become a realistic single-GPU workload.

That is the useful relationship between quantization and hardware.

Quantization does not merely make the same model use fewer gigabytes.

It can move a workload into a completely different hardware class.

A real Qwen3.6-27B example

NVIDIA's current Qwen3.6-27B-NVFP4 checkpoint is a useful example because the original model has 27B parameters.

The published NVFP4 checkpoint is about 21.9GB on Hugging Face. NVIDIA says quantizing the linear operators from 16-bit to NVFP4 reduces disk size and GPU-memory requirements by approximately 2.5× for this model.

That immediately demonstrates why the simple:

27B × 0.5 bytes = 13.5GB

calculation is insufficient.

The deployed model is not just 13.5GB of naked four-bit weights.

It contains a mixture of:

• NVFP4 operations
• higher-precision components
• scales
• multimodal components
• model metadata
• other tensors

And once it is running, it still needs memory for the serving runtime and KV cache.

That is why the KV cache guide belongs directly beside this article.

NVFP4 makes the model smaller. It does not make the rest of inference disappear.

HivenetQuant takes a mixed-precision approach

Our HivenetQuant Qwen3.6-27B-NVFP4 follows the same broader principle: use aggressive low precision where the model tolerates it and preserve more precision where it does not.

In our checkpoint, the MLP is the main NVFP4 W4A4 tier, while attention and recurrent DeltaNet layers remain at FP8. The per-layer split is chosen according to weight sensitivity rather than assuming that every layer reacts equally well to four-bit quantization.

That is an important principle for quantization generally:

uniform precision is simple. Selective precision can be better.

The largest layers often provide the biggest memory and compute savings.

A comparatively small sensitive layer may not be worth pushing into four bits if doing so harms the model disproportionately.

Why would some layers stay at FP8?

Different parts of a neural network have different error sensitivity.

Suppose two layers occupy:

Layer A = 4GB
Layer B = 200MB

and quantizing each to four bits creates a similar quality risk.

Moving Layer A to NVFP4 saves a large amount of memory.

Keeping Layer B at FP8 costs relatively little.

The economically sensible precision map may therefore be:

large tolerant layer
→ NVFP4

small sensitive layer
→ FP8

rather than forcing both to the same format.

This is why some of the most useful low-precision models are mixed-precision models.

The title on the model card may say NVFP4.

The actual engineering can contain several precisions.

Is NVFP4 accurate?

The answer cannot be “yes” without naming a model and evaluation.

NVIDIA's Qwen3.6-27B checkpoint provides a useful example.

Its published comparison includes:

‍

‍

The NVFP4 checkpoint is slightly higher on some metrics and slightly lower on others.

That is what a healthy quantization discussion should look like.

Not:

NVFP4 has no accuracy loss.

But:

On this model, using this quantization recipe, these evaluations remain close to the higher-precision reference.

Then test your own application.

“Near-lossless” needs an evaluation attached to it

Quantization claims should always answer four questions:

Which model?
An 8B dense model and a 400B MoE can react differently.

Which layers?
Weight-only quantization and W4A4 mixed precision are different.

Which benchmark?
MMLU does not tell you whether a JSON extraction service still produces valid schemas.

Which runtime?
The checkpoint format and execution path can change actual behavior.

A useful internal evaluation might measure:

• coding correctness
• reasoning benchmarks
• tool-call accuracy
• structured-output validity
• domain-specific questions
• multimodal tasks
• perplexity
• latency
• throughput

If your application cares about tool calls, a tiny benchmark change that breaks 8% of tool-call schemas is not “near-lossless” for your product.

Evaluate what matters.

Is NVFP4 faster than FP8?

It can be.

But “four bits is half of eight bits, therefore the model is twice as fast” is not a serious performance model.

Reducing precision can help by:

• reducing memory traffic
• fitting more data in caches
• reducing the model's VRAM footprint
• using Blackwell's native FP4 Tensor Cores
• allowing larger batches or more concurrent work

Actual end-to-end speed still depends on:

• model architecture
• kernel implementation
• batch size
• sequence length
• prefill versus decode workload
• GPU
• serving framework
• quantization recipe
• KV-cache configuration

A memory-bound operation may gain substantially from moving less data.

Another workload may be limited elsewhere.

Measure tokens per second and latency on the workload you intend to serve.

Do not infer them from bit width.

Smaller models can become faster for an indirect reason too

Suppose an FP8 checkpoint consumes nearly all available VRAM.

You might be forced to use:

batch size = 1

or very low concurrency.

If NVFP4 frees enough memory for:

batch size = 4

the largest performance improvement may come from the different serving configuration that the freed memory makes possible, not simply from the raw FP4 matrix operation.

This is why memory efficiency and compute performance are related.

VRAM is not passive storage.

It determines which workloads the server can keep active simultaneously.

NVFP4 and KV-cache precision are separate

This distinction is worth repeating because it causes a lot of confusion.

You can have:

NVFP4 model
+
FP8 KV cache

or another supported combination.

The precision used for the model's linear operations does not automatically determine how keys and values are stored during generation.

NVIDIA and inference frameworks now support lower-precision KV-cache options as a separate optimization path. TensorRT-LLM, for example, documents NVFP4 KV-cache support independently from model weight/activation quantization.

So if you want to know the memory requirement of an NVFP4 model at 128K context, you still need to ask:

What precision is the KV cache using?

Our KV cache guide shows how to calculate that part.

NVFP4 versus INT4

Both can use four bits.

They represent numbers differently.

INT4 represents integer levels.

NVFP4 represents a tiny floating-point format with:

sign
exponent
mantissa

and combines those values with hierarchical scaling.

Floating point makes the representable values non-uniform.

That can be useful for neural-network tensors whose values span different magnitudes.

But “floating point is better than integer” is too simplistic.

High-quality INT4 quantization methods can work very well.

The right format depends on:

• model
• hardware
• runtime
• calibration
• target accuracy
• target throughput

NVFP4's particular advantage is that it was designed together with Blackwell's native FP4 execution path.

NVFP4 versus GPTQ and AWQ

These names belong to slightly different categories.

NVFP4 is primarily a numerical format and quantization recipe.

GPTQ and AWQ are quantization methods or families of methods.

GPTQ and AWQ are commonly associated with low-bit integer model weights, but the important conceptual distinction is:

format
=
how numbers are represented

quantization method
=
how you decide which values/scales/parameters
the low-precision model should use

Those ideas can interact.

You should therefore avoid treating:

NVFP4 vs GPTQ

as if it were exactly the same kind of comparison as:

NVFP4 vs FP8

The first mixes a format/recipe with an algorithmic approach.

The second compares numerical precision formats more directly.

This distinction matters when reading model cards.

NVFP4 versus MXFP4

MXFP4 is another four-bit floating-point format supported by Blackwell.

The main scaling difference is:

MXFP4:
1 shared scale for 32 values

NVFP4:
finer FP8 scale for 16 values
+
global tensor scale

NVIDIA positions NVFP4's finer and more expressive scaling as a way to reduce quantization error compared with MXFP4.

MXFP4 can still be useful.

The correct question is not which acronym sounds more advanced.

It is which format is supported efficiently by your model, runtime, and target hardware while preserving the quality you need.

Can NVFP4 make a model fit on one GPU?

Sometimes that is exactly the point.

Take the rough 27B example again:

BF16 raw weights
≈ 54GB

A 32GB RTX 5090 cannot hold those BF16 weights.

A well-designed NVFP4 checkpoint can bring that model into the 32GB hardware class while leaving some room for runtime state and cache.

Our Qwen3.6-27B cloud GPU guide walks through that deployment pattern.

This is one of the most consequential uses of quantization.

A model does not merely become “30% cheaper.”

It can move from:

multi-GPU required

to:

single GPU possible

That changes deployment complexity as well as cost.

Quantization can be more valuable than buying another GPU

Suppose your BF16 model requires two GPUs because the weights do not fit on one.

You have at least two options:

keep BF16
+
add another GPU

or:

quantize model
+
run on one GPU

The first preserves the original precision.

The second reduces hardware and removes multi-GPU communication from the serving path.

Neither is universally correct.

But that is the right economic comparison.

Quantization is an infrastructure decision.

Its value is often measured in GPUs avoided, not merely gigabytes saved.

And sometimes you should simply use another GPU

Do not quantize a model aggressively because reducing GPU count has become an ideological goal.

A higher-memory GPU can be the better solution when:

• quality loss is unacceptable
• quantization support is immature
• the model architecture behaves poorly at four bits
• latency is more important than hardware count
• you need much more KV-cache capacity anyway
• engineering around the quantized checkpoint becomes fragile

The purpose of infrastructure is to run the workload reliably.

If FP8 on a larger-memory GPU is clearly better for the product, use FP8 on the larger GPU.

Which GPUs support NVFP4?

Native NVFP4 acceleration is associated with NVIDIA Blackwell and later hardware. NVIDIA's current CUDA math documentation requires Blackwell-or-newer compute capability for NVFP4 matrix multiplication, and RTX Blackwell adds native FP4 Tensor Core support.

For the Hivenet cluster, the most relevant example is the:

RTX 5090

with:

• Blackwell architecture
• 5th-generation Tensor Cores
• 32GB GDDR7
• native FP4 support

The exact set of supported model formats can still differ between vLLM, TensorRT-LLM, Transformer Engine, and other runtimes.

“Blackwell GPU” answers the hardware question.

It does not guarantee that every NVFP4 checkpoint works in every piece of inference software.

How do you run an NVFP4 model?

The easiest route is normally to use a pre-quantized checkpoint designed for your serving engine.

For example, NVIDIA publishes Model Optimizer NVFP4 checkpoints that can be loaded by compatible inference stacks such as vLLM and TensorRT-LLM. Its Qwen3.6 model card provides an OpenAI-compatible vLLM serving configuration.

For our Qwen deployment, we keep the full serving instructions in the Qwen3.6-27B guide rather than duplicating a version-sensitive command here.

That is intentional.

This article should explain the precision.

The model tutorial should own the deployment command.

Can you quantize a model to NVFP4 yourself?

Yes, for supported models and toolchains.

NVIDIA Model Optimizer provides post-training quantization and other model-optimization workflows, and TensorRT-LLM can consume compatible pre-quantized Model Optimizer checkpoints. NVIDIA's current tooling supports NVFP4 alongside FP8 and several integer quantization recipes.

But quantizing a model well involves more than running:

--quantize-to-4-bits

A serious workflow can involve:

• calibration data
• per-layer sensitivity
• deciding which modules to exclude
• activation quantization
• KV-cache precision
• accuracy evaluation
• runtime compatibility
• kernel support

That is why publishing a quantized model should include a model card explaining what was quantized and how the result was evaluated.

A four-bit file with no methodology is difficult to trust.

What should a good NVFP4 model card tell you?

At minimum, I would want:

Base model
Which checkpoint was quantized?

Quantization format
Is it W4A4, W4A16, or a mixed configuration?

Layer policy
Which modules use NVFP4, FP8, or BF16?

Target hardware
Is it intended for Blackwell?

Runtime
vLLM? TensorRT-LLM? Something else?

Calibration
What data or method was used?

Memory footprint
How large is the actual checkpoint and deployment?

Evaluation
How does it compare with the reference model?

Limitations
Which tasks or configurations are known to degrade?

That information is far more useful than:

Quantized to four bits for blazing-fast inference.

The second sentence tells you almost nothing.

Is NVFP4 only for LLMs?

No.

FP4 inference applies more broadly to neural-network workloads that can use compatible low-precision operations.

NVIDIA supports FP4 paths for visual generative AI as well, including models such as FLUX on Blackwell GPUs. Its current visual-generation support matrix includes FP4 and FP8 configurations for the RTX 5090.

We touched this from the application side in our FLUX.1 [dev] guide.

The same underlying hardware idea applies:

reduce precision where the model tolerates it so more of the workload fits and moves efficiently through the GPU.

What does NVFP4 change for inference economics?

Suppose quantization lets you move from:

2 GPUs

to:

1 GPU

for the same model.

The benefit is larger than the reduction in checkpoint bytes.

You may also remove:

• tensor-parallel communication
• another GPU's hourly cost
• another source of synchronization overhead
• some multi-GPU configuration complexity

Or suppose the model already fits on one card.

Reducing weight memory can instead leave room for:

• longer contexts
• more KV cache
• greater concurrency
• larger batches
• additional model components

Quantization gives you a memory budget.

What you do with that budget determines the real value.

When should you use NVFP4?

NVFP4 is particularly interesting when all of these are true:

You are running Blackwell hardware.

The native hardware path is a major reason to use the format.

Model weights are a meaningful memory constraint.

If an 8B model already occupies a small fraction of your GPU, aggressive quantization may solve a problem you do not have.

A high-quality NVFP4 checkpoint exists or you can validate one.

The format alone does not guarantee quality.

Your serving framework supports the model correctly.

A theoretically efficient format is useless if your runtime falls back to a slow path or cannot load it.

The evaluation passes.

This is the deciding test.

When should you avoid NVFP4?

I would not choose it automatically when:

• the target GPU predates Blackwell
• the model comfortably fits at FP8 or BF16
• four-bit quantization causes meaningful task degradation
• the serving stack has immature support
• the model relies heavily on layers that cannot be quantized safely
• a higher-precision configuration already meets the cost target

“Lower precision” is not a product requirement.

It is one tool for meeting memory, performance, and cost requirements.

NVFP4 does not replace good model selection

A badly chosen 70B model quantized brilliantly can still be a worse system than a strong 27B model running at higher precision.

Model size, precision, and GPU count should be chosen together.

A useful decision order is:

1. Which model passes the task evaluation?

2. Which precision preserves that result?

3. How much memory does the resulting checkpoint need?

4. How much context and concurrency do we need?

5. Which GPU configuration serves it efficiently?

Do not reverse that process and choose the model because you have found an interesting quantization format.

The useful way to think about NVFP4

NVFP4 is best understood as a hardware-aware way of spending precision.

Four-bit values make the model smaller.

Fine-grained scales recover much of the numerical flexibility that naive four-bit representation would lose.

Mixed precision lets sensitive parts remain at FP8 or BF16.

Blackwell Tensor Cores make the resulting low-precision operations useful for real inference rather than merely compressed storage.

The result can be substantial.

A model that previously needed several GPUs can sometimes move onto one.

A model already on one GPU can leave far more memory available for cache and concurrent requests.

But none of that means four bits are magically equivalent to sixteen.

The correct question remains:

Did this particular quantized model preserve the behavior we need while improving the infrastructure enough to matter?

If the answer is yes, NVFP4 can be a very powerful trade.

NVFP4 FAQ

What is NVFP4?

NVFP4 is NVIDIA's 4-bit floating-point format for Blackwell GPUs. It combines E2M1 four-bit values with fine-grained FP8 block scaling and a global scaling factor to reduce quantization error at low precision.

What does FP4 mean?

FP4 means a four-bit floating-point representation. In NVFP4, the core value uses E2M1: one sign bit, two exponent bits, and one mantissa bit.

What does NVFP4 E2M1 mean?

E2M1 describes the bit layout of the four-bit value: two exponent bits and one mantissa bit, plus a sign bit. The raw format can represent magnitudes up to 6 before scaling is applied.

What is the NVFP4 block size?

NVFP4 uses fine-grained block scaling. The standard one-dimensional form shares an FP8 E4M3 scale across 16 consecutive values. NVIDIA's Transformer Engine also supports 16 × 16 two-dimensional scaling for weight tensors.

What does W4A4 mean?

W4A4 means four-bit weights and four-bit activations. An NVFP4 W4A4 model performs compatible matrix operations using low-precision values for both.

Is NVFP4 the same as INT4?

No. INT4 represents four-bit integers. NVFP4 uses four-bit floating-point E2M1 values combined with hierarchical scaling.

Is NVFP4 the same as GPTQ?

No. NVFP4 describes a numerical format and quantization recipe, while GPTQ is a quantization method. A model card can therefore involve both a numerical format and an algorithm used to decide how quantization is applied.

What is the difference between NVFP4 and FP8?

FP8 uses eight bits for the main numerical value, while NVFP4 uses four plus fine-grained scaling. NVFP4 can use substantially less memory, while FP8 provides greater intrinsic numerical resolution.

How much memory does NVFP4 save?

NVIDIA reports roughly 3.5× lower model memory than FP16 and 1.8× lower than FP8 in its published NVFP4 analysis. Real savings vary because models often retain some layers and metadata at higher precision.

Does NVFP4 reduce accuracy?

It can. The amount depends on the model and quantization recipe. NVIDIA's published Qwen3.6-27B NVFP4 evaluation remains close to its FP8 reference across the benchmarks it reports, but that result should not be generalized to every model.

Is NVFP4 faster than FP8?

It can improve inference performance on supported Blackwell hardware by reducing memory traffic and using native FP4 Tensor Core operations. Actual end-to-end speed depends on model architecture, kernels, batch size, context, and serving framework.

Which GPUs support NVFP4?

Native NVFP4 execution requires NVIDIA Blackwell-class or newer hardware in supported software stacks. The RTX 5090 is a Blackwell GPU with fifth-generation Tensor Cores and native FP4 support.

Does the RTX 5090 support NVFP4?

Yes. The RTX 5090 uses the Blackwell architecture and its fifth-generation Tensor Cores support FP4 operations. It has 32GB of GDDR7 VRAM.

Can NVFP4 make a 27B model fit on an RTX 5090?

Yes, depending on the model and quantization recipe. A 27B BF16 model requires roughly 54GB for raw weights, while existing NVFP4 Qwen3.6-27B checkpoints fit within the 32GB hardware class.

Is an NVFP4 model exactly one-quarter the size of BF16?

No. Four-bit values theoretically require one quarter of the raw storage of 16-bit values, but real checkpoints include scales, metadata, and components left at higher precision. NVIDIA reports real reductions below the theoretical 4× maximum.

Does an NVFP4 model use an NVFP4 KV cache?

Not automatically. Model precision and KV-cache precision are separate configuration choices. Some serving stacks support FP8 or NVFP4 cache formats independently.

Can vLLM run NVFP4 models?

Yes, for compatible checkpoints and quantization formats. NVIDIA publishes NVFP4 models intended for vLLM deployment, including Qwen3.6-27B.

Can I quantize my own model to NVFP4?

Yes, for supported models and environments. NVIDIA Model Optimizer provides post-training quantization workflows and exports checkpoints for compatible serving frameworks.

Is NVFP4 only useful for very large models?

No, but its value becomes easier to justify as model memory, bandwidth, or concurrency becomes a real constraint. If the model already fits comfortably at higher precision, the benefit may be smaller.

‍