LLM quantization guide: methods, memory, and evaluation

LLM quantization reduces the precision used to store or process selected model tensors. The immediate benefit is usually lower memory use. Depending on the model, format, runtime, and hardware, it can also improve throughput or latency. Those performance gains are not automatic, and a smaller checkpoint is only useful if it preserves the capabilities your workload needs.

This guide covers the complete deployment decision: what can be quantized, how the common methods differ, how to estimate memory, and how to evaluate a quantized model before production. If your decision is specifically between two integer precisions, use our focused INT4 vs INT8 comparison.

What LLM quantization can change

“A 4-bit model” is incomplete shorthand. A deployment may reduce the precision of weights, activations, the KV cache, or some combination of them. It may also store values at one precision and perform arithmetic at another.

 
• Weight-only quantization reduces the largest static part of many LLM checkpoints. Labels such as W4A16 mean 4-bit weights with 16-bit activations.
 
• Weight-and-activation quantization can reduce memory traffic and accelerate matrix operations when the hardware and kernels support the chosen format. W8A8 means 8-bit weights and 8-bit activations.
 
• KV-cache quantization targets the memory that grows with sequence length and concurrent requests. It is a separate choice from weight precision and needs its own quality test.
 
• Mixed precision leaves sensitive layers or operations at higher precision while compressing the parts that tolerate it.

Quantization does not change tokenization or the public shape of an inference API. It changes how tensors are represented and how the runtime executes the model.

For the broader toolkit beyond quantization, compare LLM optimization techniques including fine-tuning, pruning, sparsity, and knowledge distillation.

Integer and floating-point formats are different

Bit width alone does not identify a quantization format. INT4, NF4, FP4, and NVFP4 all use four bits, but they represent values differently and depend on different software and hardware paths.

 
• INT8 offers more representable integer values than INT4 and is often used for weight-only or weight-and-activation quantization.
 
• INT4 compresses more aggressively. Common inference recipes keep activations at a higher precision, such as W4A16.
 
• NF4 and FP4 are four-bit floating formats. Hugging Face documents NF4 as a format adapted to normally distributed weights and commonly used in 4-bit fine-tuning workflows.
 
• FP8 and NVFP4 have hardware-specific execution paths. NVIDIA documents NVFP4 and several FP8, AWQ, and GPTQ recipes in TensorRT-LLM.

Check the checkpoint configuration and runtime documentation instead of inferring behavior from “4-bit” or “8-bit” in a model name.

Common quantization methods

Most teams start with a pre-quantized checkpoint or post-training quantization rather than retraining a model from scratch.

 
• GPTQ is a one-shot post-training method that uses approximate second-order information to reduce weight precision.
 
• AWQ uses activation statistics from calibration data to protect important weight channels during low-bit weight quantization.
 
• SmoothQuant is a training-free method designed for 8-bit weights and activations. It shifts quantization difficulty between activations and weights to make W8A8 serving more practical.
 
• LLM.int8() handles activation outliers at higher precision while using 8-bit computation for the remaining values.
 
• Quantization-aware training simulates reduced precision during training. It can improve results in some cases, but costs more time and compute than a post-training workflow.

The method name is only one variable. Group size, scale granularity, calibration data, skipped modules, compute type, and serving kernels can all change memory use, speed, and quality.

Estimate weight memory before choosing hardware

A useful first-pass estimate is:

weight memory ≈ parameter count × bits per weight ÷ 8

Using that raw calculation:

 
• 8B parameters: about 16 GB at 16-bit, 8 GB at 8-bit, and 4 GB at 4-bit.
 
• 27B parameters: about 54 GB at 16-bit, 27 GB at 8-bit, and 13.5 GB at 4-bit.
 
• 70B parameters: about 140 GB at 16-bit, 70 GB at 8-bit, and 35 GB at 4-bit.

These are weight-only estimates, not complete VRAM requirements. Scales, zero points, packing metadata, unquantized layers, activations, runtime buffers, and the serving engine add overhead. The KV cache also grows with context length and concurrency. Training and fine-tuning require additional memory for saved activations, gradients, and optimizer state, which is why the same model may fit for inference and still exceed the available hardware during training. Our training vs inference hardware guide explains that distinction.

Runtime and hardware support come before the format

A checkpoint is deployable only when the runtime can execute its quantization method on the target processor. Current vLLM quantization documentation lists support by method and hardware family. NVIDIA publishes a separate TensorRT-LLM support matrix for its INT, FP8, and FP4 recipes.

Those matrices change as runtimes add kernels. Confirm the exact runtime version, model architecture, GPU generation, and quantization configuration at deployment time. A format that is compact on disk may fall back to higher precision for unsupported operations or spend enough time packing and dequantizing values that it produces little speed benefit.

Quantization often raises capacity before it raises speed. Lower weight memory can let you fit the model on fewer GPUs, increase batch size, or leave more room for the KV cache. Prefill may remain compute-bound while decode benefits from lower memory traffic. Measure both phases rather than reporting one universal speed number.

How to choose a quantized checkpoint

 
1. Start with the deployment stack. List the target GPU or accelerator, runtime version, model architecture, context length, concurrency, and latency target.
 
2. Verify checkpoint provenance. Confirm the base model revision, license, chat template, quantization method, compute type, calibration notes, and supported runtime.
 
3. Use the least aggressive precision that solves the constraint. If an 8-bit or FP8 model fits and meets the target, a 4-bit path may add testing and compatibility work without a useful payoff.
 
4. Match calibration data to the workload. Code, multilingual prompts, retrieval, tool use, and long-context tasks can react differently to the same recipe.
 
5. Keep a higher-precision baseline. You need a stable reference to determine what the optimization changed.

Evaluate quality and performance on the same workload

Generic leaderboard results are useful screening evidence, but the release decision should come from the model, prompts, runtime, and hardware you will use.

 
1. Build an evaluation set from real prompts and failure cases. Keep a held-out set that was not used for calibration.
 
2. Run the higher-precision baseline and every quantized candidate with the same chat template, decoding settings, context limits, and output limits.
 
3. Measure task quality. Use workload-specific checks for factuality, instruction following, retrieval, structured output, code, tool calls, multilingual behavior, or safety.
 
4. Measure peak VRAM, time to first token, inter-token latency, output tokens per second, throughput at target concurrency, and error rates.
 
5. Test short and long contexts separately. KV-cache pressure can change the result as sequence length and batch size grow.
 
6. Record the runtime, driver, kernel, model revision, and quantization configuration so the comparison can be reproduced.

Roll out with a fallback

Send a small share of traffic to the quantized candidate, compare outputs and operating metrics with the baseline, and keep a fast route back to the higher-precision model. Monitor quality samples alongside latency, throughput, GPU memory, and failure rates. A model that passes an offline benchmark can still behave differently when prompts, context lengths, or traffic shape change.

HivenetQuant: publish the trade-off, not just the checkpoint

HivenetQuant publishes optimized open-model checkpoints with the measurements used to evaluate them. Its current work includes NVFP4 mixed-precision models for Blackwell GPUs and reports task-level regressions as well as performance gains. That evidence matters because an average score can hide a larger change in the workload you care about.

If you need hardware for your own comparison, review Hivenet’s current GPU and CPU rental options and benchmark the candidate with your actual serving stack.

Frequently asked questions

Does quantization always make an LLM faster?

No. It reduces representation size first. Speed depends on hardware support, kernels, model architecture, batch shape, context length, and whether the workload is limited by compute or memory movement.

Does a 4-bit model use exactly half the memory of an 8-bit model?

Raw weight storage is roughly half, but complete VRAM use is not. Scales, metadata, higher-precision layers, activations, the KV cache, and runtime buffers reduce the real difference.

Do I need to retrain the model?

Usually not for post-training methods such as GPTQ, AWQ, or SmoothQuant. Calibration and reconstruction may still be required. Quantization-aware training is a separate, more expensive path.

Can the KV cache be quantized?

Some runtimes support lower-precision KV caches, but compatibility and quality effects vary. Treat KV-cache precision as a separate experiment from weight quantization.

How do I know whether quality dropped?

Compare the quantized model with a higher-precision baseline on representative and held-out tasks. Review task-level results, not only an average score.

Primary sources and further reading

 
• vLLM quantization formats and hardware compatibility
 
• Hugging Face bitsandbytes documentation
 
• NVIDIA TensorRT-LLM quantization documentation
 
• GPTQ paper
 
• AWQ paper
 
• SmoothQuant paper

‍