A practical guide to LLM quantization

Most inference problems are memory problems. Quantization shrinks model weights so you can fit the model and its cache on the GPUs you have, batch deeper, and keep latency steady. The trick is keeping quality in range for your tasks. Additionally, using quantization can lead to a lower carbon footprint for training and inference due to reduced power consumption.

This article is a structured, practical resource distinct from a typical blog, focusing on providing systematic information and actionable steps. Readers should be familiar with basic concepts of model inference and quantization before proceeding. In this article, we will cover prerequisites, quantization methods, and evaluation to ensure a comprehensive understanding of the topic.

Try Compute today

On Compute, you can launch a vLLM server and choose smaller, quantized model variants from the catalog. Set context and output caps, then measure TTFT and tokens/second with your own prompts.

What quantization does

Quantization stores weights with fewer bits than FP16/BF16. The number of bits used directly impacts both memory consumption and model accuracy. The model runs with lightweight de/quantization kernels so math stays stable enough for most tasks. However, standard uniform quantization can severely impact the representation of outlier weights and activations, degrading accuracy. Choosing the optimal bit-width and calibration strategy during quantization requires extensive testing to balance memory savings and accuracy.

• int8: good default for production—high win on memory with minimal quality loss. The range of values that can be stored is determined by the bit-width, and zero points are represented as specific integer values within this range. The max value in the data range is used for scaling during quantization.
• int4: bigger memory win, often faster decode at high concurrency, higher risk of quality dips on complex reasoning or long outputs. The range of values and how zero points are represented are similarly defined by the bit-width, with the max value again used for scaling.
• Note: Actual memory savings and the number of parameters affected can vary depending on the implementation and model architecture.

Quantization does not change tokenization or your API. It changes memory use and throughput. Quantization is a method for reducing model size and improving efficiency by mapping floating-point values to a smaller set of discrete values.

Common methods in the wild

• AWQ (Activation‑aware Weight Quantization). This method is applied in various production settings and tunes quantization using real activations so important channels keep precision. Improved memory efficiency and strong int4 results for many chat models are typically achieved.
• GPTQ. Post‑training per‑channel quantization with calibration data. This applied method is widely available; results achieved vary by group size and settings.
• LLM.int8 / bitsandbytes. Popular int8 path that preserves outlier weights. Reliable when you want a quick, safe reduction.
• Marlin‑style kernels. Optimized GPU kernels that speed up low‑bit matmuls on supported cards.
• Quantization-Aware Training (QAT). Advanced techniques like QAT aim to minimize accuracy loss but require more computational resources.

Pick what your serving stack supports and what your model family offers pre‑built. Avoid one‑off toolchains unless you plan to maintain them.

Memory math you can do on a napkin

Baseline weight size for FP16 is ~2 bytes per parameter.

• 7B model, FP16: ~14 GB for weights
• 7B, int8: ~7–8 GB
• 7B, int4: ~3.5–4 GB

Smaller LLMs are generally more sensitive to information loss during quantization compared to larger models.

Add KV‑cache headroom: roughly hidden_size × num_layers × 2 (K/V) × seq_len × batch in bytes at runtime (precision depends on the engine). If cache pressure climbs, TTFT rises and tokens/second falls.

Start in seconds with the fastest, most affordable cloud GPU clusters.

Launch an instance in under a minute. Enjoy flexible pricing, powerful hardware, and 24/7 support. Scale as you grow—no long-term commitment needed.

Try Compute now

Speed and throughput

The focus of this section is on throughput and batching. Quantization can raise throughput because you can batch more requests before memory runs out. Additionally, quantization can improve throughput and efficiency in serving models by reducing memory usage and computational requirements. Prefill can still be compute‑bound, so gains vary by model, prompt length, and kernels. Measure on your prompts. Don’t promise speed without data. It is often necessary to assess trade-offs when deciding to use quantized models based on their use cases.

When it helps

• Quantization may suit users who face specific memory or cost constraints, such as when:
• You routinely hit VRAM limits or cache eviction under load.
• You want larger batch sizes at the same latency target.
• You must fit a model on fewer GPUs to cut cost.

When it hurts

• Long, multi‑step reasoning with strict accuracy targets.
• Safety or classification tasks that are sensitive to small score shifts.
• Very long outputs where errors compound.

Example applications

Quantization and KV caching aren't just trendy techniques—they're foundational tools that make language models work efficiently without sacrificing quality. Take transformer architectures like GPT: KV caching lets them handle longer input sequences while using less power and memory per inference. The usability of KV caching is particularly pronounced for AI models generating longer texts, as it helps maintain efficiency and performance. When you're deploying on devices with tight resource constraints, every byte and millisecond matters. Key-Value caching helps speed up text generation in AI models by remembering important information from previous steps.

Quantization cuts your model's memory footprint by reducing weight precision. You get faster inference while keeping text quality high. Post-training methods like GPTQ let you deploy large language models without retraining—perfect when you need that sweet spot between performance and resource use. Post-Training Quantization (PTQ) quantizes an already trained model and is faster to implement but may significantly degrade accuracy. NLP applications demand coherent, contextually accurate text, and your models need to work across different devices and environments. The calibration process is necessary to find the min and max values for quantization.

Building efficient models means understanding how quantization affects accuracy and how KV caching reduces computational costs. You'll want clear code examples and tutorials that show the implementation process. Compare int8 and int4 quantized models using tables or diagrams—this helps you see the memory, speed, and quality trade-offs. Pick the approach that fits your application's needs. Regular large language models require significant hardware resources proportional to their size.

Making language models efficient comes with real challenges. You need to maintain output quality across diverse topics and input lengths. Traditional hardware has limits. Your deployed models must generate reliable results when real users hit them with real-world inputs. Stay current with research papers, articles, and implementation guides—they'll help you make smart decisions and improve your models' efficiency.

Quantization and KV caching deliver measurable impact on language model performance and efficiency. Focus on these techniques and you can deploy powerful NLP solutions that work across many use cases. Keep memory usage, inference costs, and deployment complexity under control.

A simple evaluation loop

1. Pick 30–100 real prompts that reflect your product. Include hard cases.
2. Define checks: automatic metrics (exact match, BLEU/ROUGE if relevant), plus quick human review for faithfulness. Evaluation methods are applied to gather evidence of model performance, and human feedback is valuable for assessing faithfulness and quality.
3. Run FP16 baseline on your target hardware. Record TTFT, tokens/second, and any critical task scores.
4. Test int8, then int4 on the same hardware and settings. Keep context/output caps identical.
5. Compare deltas: quality, TTFT, tokens/second, and GPU memory headroom. The differences in quality and performance are determined by comparing these metrics.
6. Decide: ship int8 if quality is within tolerance; consider int4 only if quality holds on your tasks.

Results can be presented in tables or charts for clarity.

Rollout plan that avoids surprises

• Shadow traffic for a subset of users.
• Guardrails: cap max_tokens, keep repetition penalties and stop sequences consistent.
• Fast rollback via feature flag or gateway route. Feature flags can be enabled or disabled to control rollout and quickly revert changes if needed.
• Dashboards for TTFT/TPS, error rates, and quality samples. Dashboards and monitoring tools can be integrated into the rollout process to provide better visibility and ensure smooth operations.

Troubleshooting

• Outputs look terse or generic. This section helps tackle common quantization issues. Raise max_tokens slightly; check for over‑aggressive group sizes in int4 models.
• Latency improved but spikes under load. Cache is tight. Trim prompts, reduce caps, or add VRAM.
• Quality wobbles on niche tasks. Keep that path on FP16 or try int8 with outlier handling to overcome quality drops.
• OOM on long chats. Shorten history, use RAG, or move to a bigger preset to overcome OOM errors.

Last thoughts

Quantization is one of the cleanest ways to fit models, keep queues healthy, and control spend. Start with int8, measure on your data, and move to int4 only when the numbers say it is safe.

Understanding the word 'quantization' is key to making informed decisions about model optimization and deployment.

For more technical details and in-depth explanations, consult the references provided by authoritative sources.

Try Compute today

Launch a quantized model on a vLLM endpoint in Compute, keep your OpenAI client, and compare TTFT and tokens/second against your baseline before you roll out.

FAQ

What is quantization in LLMs?

Storing and computing with fewer bits for model weights (and sometimes activations) to cut memory use and raise throughput.

Is 4‑bit good enough?

Often for casual chat and summarization. Test carefully for reasoning, tool use, and long outputs. When in doubt, start with int8.

Does quantization always speed things up?

No. It raises capacity first by reducing memory. Speedups depend on kernels, batch shape, and prompt length.

What about the KV‑cache—can that be quantized?

Some stacks support lower‑precision KV‑cache. Gains vary and may affect quality. Treat as an advanced option after weight quantization proves safe.

Do I need to retrain the model?

Not for post‑training methods like AWQ and GPTQ. You run a calibration step at most.

Will prompts or tokenization change?

No. Quantization is an internal representation detail.

How do I tell if quality dropped?

Use a small eval set and a quick human pass. Watch for loss of structure, missed steps, and factual drift.

‍