INT4 vs INT8 quantization for AI inference

Choose INT8 when the model fits, your runtime supports an efficient 8-bit path, and quality stability matters more than maximum compression. Choose INT4 when weight memory is the binding constraint and a validated 4-bit checkpoint can move the workload onto fewer GPUs or leave enough VRAM for context and concurrency.

That is the practical answer. Neither precision guarantees a speedup or a fixed accuracy loss. The result depends on what is quantized, the model and calibration method, the runtime kernels, the target hardware, and the workload used for evaluation. For the broader workflow and method definitions, start with our LLM quantization guide.

INT4 vs INT8 at a glance

 
• Raw weight storage: INT8 uses about one byte per weight; INT4 uses about half a byte before scales, metadata, and other overhead.
 
• Compression from FP16: INT8 weight storage is roughly half the raw FP16 size. INT4 is roughly one quarter.
 
• Representable values: an integer stored in eight bits has many more levels than one stored in four bits, so INT8 is usually less sensitive to scaling and outliers.
 
• Common inference recipes: INT8 may quantize weights and activations, while many INT4 deployments use 4-bit weights with 16-bit activations, written W4A16.
 
• Operational risk: INT4 depends more heavily on the checkpoint recipe, calibration, group size, packing layout, and specialized kernels.

Do not treat every 4-bit model as INT4

INT4 is an integer format. NF4, FP4, and NVFP4 are four-bit floating-point formats with different value representations and execution paths. A model labeled “4-bit” may use any of these formats or a method-specific packed representation.

The distinction affects compatibility. Hugging Face documents FP4 and NF4 in its bitsandbytes integration. NVIDIA documents NVFP4 and W4A16 AWQ or GPTQ as separate TensorRT-LLM recipes. Read the model configuration and runtime documentation before selecting hardware.

Raw memory savings

For weights alone, use parameter count × bits ÷ 8:

 
• 8B parameters: about 8 GB at INT8 and 4 GB at INT4.
 
• 27B parameters: about 27 GB at INT8 and 13.5 GB at INT4.
 
• 70B parameters: about 70 GB at INT8 and 35 GB at INT4.

Actual VRAM use is higher. Quantized checkpoints also store scales, zero points, and packing metadata, and some layers may stay at higher precision. Activations, the KV cache, runtime workspaces, and the serving engine consume additional memory. INT4 can cross an important fit boundary, but it does not make total memory exactly half of an INT8 deployment.

Estimate the full workload with context length and target concurrency, then compare it with the GPUs available in the GPU guide for LLM inference.

Which one is faster?

There is no defensible universal percentage. Lower precision can reduce data movement and make more efficient matrix operations available, but only when the hardware and runtime have kernels for the exact format.

INT4 has the larger theoretical bandwidth advantage. It can still underperform when values must be unpacked inefficiently, operators fall back to higher precision, or the workload is limited by a part of the system that was not quantized. INT8 often has a more mature execution path, but support also varies by GPU generation and runtime version.

Measure prefill and decode separately. Prompt processing may remain compute-bound, while token-by-token decode benefits more from lower weight memory traffic. Batch size and concurrency can change the comparison again.

Which one preserves quality better?

INT8 usually provides more numerical headroom, so it is the safer starting point when both candidates use comparable methods and the 8-bit model fits. INT4 is more sensitive to calibration data, scaling granularity, outlier handling, and which layers remain at higher precision.

A fixed “INT4 loses X percent” rule is misleading. Quality changes by model, task, method, and evaluation. Coding, retrieval, multilingual prompts, tool use, long context, structured output, and reasoning can respond differently. Compare task-level results with the higher-precision baseline and inspect the failures that matter to the application.

Hardware and runtime support decide what is practical

Current vLLM documentation lists support separately for AWQ, GPTQ, W8A8 INT8, bitsandbytes, GGUF, and other paths. The TensorRT-LLM matrix separates W4A16, W4A8, FP8, and FP4 recipes by model and GPU architecture.

Check those matrices for the version you will deploy. “The GPU supports integer operations” is not enough. The model architecture, quantization recipe, runtime, kernel, and GPU generation must work together.

Choose INT8 when

 
• The model and required KV-cache headroom already fit in available VRAM.
 
• The runtime has a documented, efficient INT8 path for the target hardware.
 
• The workload has a narrow quality tolerance or limited evaluation coverage.
 
• You need a simpler baseline for debugging and reproducibility.
 
• INT4 does not cross a useful hardware, context, or concurrency boundary.

Choose INT4 when

 
• The INT8 model does not fit with the required context and concurrency.
 
• Four-bit weights let you use fewer GPUs or a simpler serving topology.
 
• A supported checkpoint and kernel exist for the exact model and hardware.
 
• You can evaluate the model on representative and held-out tasks.
 
• The measured quality change stays inside the application’s tolerance.

A fair INT4 vs INT8 test

 
1. Use the same base model revision, tokenizer, chat template, prompts, and decoding settings.
 
2. Run both candidates on the same hardware and runtime version.
 
3. Record complete VRAM use, not only checkpoint file size.
 
4. Measure time to first token, inter-token latency, output tokens per second, and throughput at target concurrency.
 
5. Score the tasks the product relies on and review representative outputs.
 
6. Test short and long contexts, including KV-cache pressure.
 
7. Keep a higher-precision baseline and a rollback path.

What HivenetQuant’s four-bit work shows

HivenetQuant currently publishes NVFP4 mixed-precision checkpoints for Blackwell GPUs. NVFP4 is not INT4, but the evaluation lesson applies to both: report the exact format and serving configuration, compare with the original model, and publish task-level regressions alongside performance gains.

HivenetQuant’s results also show why one average number is insufficient. Different capabilities can react differently to the same compression recipe. The useful decision is the efficiency gained for the capability your workload gives up.

To run your own comparison, review Hivenet’s current GPU and CPU rental options and benchmark both candidates with the serving stack you intend to operate.

Frequently asked questions

Is INT4 always faster than INT8?

No. INT4 moves less weight data, but speed depends on kernels, fallbacks, batching, context length, and the workload bottleneck.

Does INT8 always preserve accuracy?

No. INT8 is usually less aggressive, but method, calibration, outliers, model architecture, and task still matter.

Can INT4 fit a model that INT8 cannot?

Often, because raw INT4 weight storage is roughly half of INT8. Confirm total VRAM after adding metadata, higher-precision modules, activations, runtime buffers, and the KV cache.

Are NF4 and NVFP4 the same as INT4?

No. They share the four-bit width but use floating-point representations and different software or hardware paths.

Should I test INT8 first?

Usually, when it fits and the runtime supports it. Test INT4 when it solves a real memory, hardware, or concurrency constraint.

Can I switch later?

Yes, but treat the new checkpoint as a separate release. Repeat quality, performance, compatibility, and rollout checks.

Primary sources and further reading

 
• vLLM quantization formats and hardware compatibility
 
• Hugging Face bitsandbytes documentation
 
• NVIDIA TensorRT-LLM quantization documentation
 
• GPTQ paper
 
• AWQ paper
 
• SmoothQuant paper

‍