KV cache explained: why longer context uses so much GPU memory

A language model can fit into GPU memory and still run out of VRAM when you give it a long prompt.

The reason is often the KV cache.

During autoregressive inference, an LLM generates one token at a time. At each attention layer, it creates key and value representations for the tokens it has already processed. Keeping those representations in memory means the model can reuse them instead of recomputing the same attention state every time it generates another token. Hugging Face describes the cache as storing the key-value pairs from previously processed tokens and appending new ones as generation continues.

That saves compute.

It also means every active token consumes memory.

For a standard transformer, KV-cache memory grows roughly linearly with:

context length × number of layers × KV heads × head dimension × precision × concurrent sequences

This is why a model advertised with a 128K or 256K context window does not automatically make that context practical on every GPU.

A context window is a model capability.

The KV cache is part of the bill you pay to use it.

What is a KV cache?

KV stands for key-value.

In transformer attention, each token is projected into three kinds of vectors:

Query
Key
Value

The query represents what the current token is looking for.

The keys help determine which previous tokens are relevant.

The values contain the information that attention retrieves from those tokens.

A simplified attention calculation looks like:

Attention(Q, K, V)

When generating text, the model repeatedly needs the keys and values associated with all the tokens it has already processed.

Without caching, generating token 1,001 would involve recalculating information the model already computed while producing token 1,000.

Then token 1,002 would repeat much of that work again.

A KV cache keeps the old keys and values around.

The new generation step only has to calculate the new token's state and attend over the stored cache. Hugging Face's current Transformers documentation describes the cache as one key tensor and one value tensor per attention layer, growing as additional tokens are processed.

The query is not stored for future decoding in the same way.

That is why we call it a KV cache rather than a QKV cache.

Why does KV cache make generation faster?

Consider a conversation containing 10,000 tokens.

The model needs those 10,000 tokens as context before predicting the next one.

Without a cache, it would repeatedly recompute key and value states for the previous context during each decoding step.

With a KV cache, those states have already been calculated.

The model keeps them and adds one new set of keys and values when the next token arrives.

Conceptually:

Prompt
1 2 3 4 5 ... 10,000
               ↓
            cached K,V

Generate token 10,001
               ↓
         add new K,V

Generate token 10,002
               ↓
         add new K,V

This is one of the reasons autoregressive inference can remain usable as output grows. Hugging Face notes that the cache avoids recomputing previous K and V states, while cache memory itself grows with sequence length.

The tradeoff is straightforward:

you spend memory to save computation.

How to calculate KV-cache memory

For a conventional decoder-only transformer with full attention, a useful first-order calculation is:

KV cache bytes =
tokens
× layers
× 2
× KV heads
× head dimension
× bytes per element

The 2 represents the two cached tensors:

K + V

For several independent sequences, multiply by the number of active sequences if they are the same length.

A more complete expression is:

KV cache bytes =
batch size
× sequence length
× number of layers
× 2
× number of KV heads
× head dimension
× bytes per element

This follows directly from the cache tensor structure used by transformer implementations: batch, heads, sequence length, and head dimension, with separate tensors for keys and values.

The formula is extremely useful.

It is also deliberately simplified.

Sliding-window attention, hybrid architectures, Multi-Head Latent Attention, cache compression, tensor parallelism, and other designs can change the real memory layout.

For a standard full-attention model, though, it gives you the right intuition.

How many bytes does each precision use?

For the simple calculation:

‍

‍

So moving a compatible cache from BF16 to FP8 roughly halves its raw storage requirement.

Notice that this is the cache precision.

It does not have to be the same thing as the model-weight precision.

You can have:

4-bit model weights
+
BF16 KV cache

or:

4-bit model weights
+
FP8 KV cache

These are separate memory decisions.

That distinction will matter again in our planned NVFP4 and Blackwell inference guide.

A worked example

Take a representative GQA transformer with:

‍

‍

For one token:

32
× 2
× 8
× 128
× 2 bytes
=
131,072 bytes

That is:

128 KiB per token

It sounds small.

Now multiply it by context length:

That is for one sequence.

‍

‍

If the model weights themselves already occupy a substantial part of a 32GB GPU, a 128K BF16 cache can make the advertised maximum context completely unrealistic for that configuration.

This is why our RTX 5090 VRAM guide repeatedly distinguishes between making model weights fit and leaving enough memory for the workload.

Larger models can have larger caches too

Parameter count does not directly determine KV-cache size.

The architecture does.

Imagine a larger GQA model with:

80 layers
8 KV heads
128 dimensions per head
BF16 cache

Its cache requires:

80
× 2
× 8
× 128
× 2
=
327,680 bytes

or:

320 KiB per token

Now the same contexts look like:

‍

‍

Again, this is a representative full-attention configuration rather than a universal table for every 70B model.

It shows why the phrase:

70B model at 4-bit needs about 35GB

is incomplete infrastructure advice.

That number describes a rough weight budget.

The running endpoint also needs a cache.

Our Llama 3.3 70B GPU requirements guide covers the multi-GPU side of that problem.

Prompt tokens consume KV cache too

KV-cache sizing is sometimes discussed as if only generated tokens matter.

They don't.

The prompt goes through a prefill phase before decoding begins. The keys and values created for those prompt tokens become the cache the model uses while generating the answer.

Suppose you send:

24,000 prompt tokens

and allow:

8,000 output tokens

The sequence can eventually contain:

32,000 active tokens

The cache has to represent the relevant attention state for that active sequence.

This is why a long document can consume considerable VRAM before the model has generated much text at all.

It also explains why the nominal context limit is generally a combined budget for input and output rather than an independent allowance for each. Meta, for example, lists 128K context for both Llama 3.1 and Llama 3.3, with GQA used to improve inference scalability.

Generated tokens keep growing the cache

Suppose instead that your prompt is short:

4K prompt

but you are running a reasoning workload that generates:

28K additional tokens

You end up in the same broad place:

32K active sequence

The cache does not care that most of those tokens came from output rather than input.

This makes output limits an infrastructure setting too.

A server configured for 2,000-token answers has a very different worst-case cache profile from one that allows 30,000-token generations.

Long reasoning is not free simply because the prompt was short.

Concurrency multiplies the problem

One request is rarely the production problem.

Suppose our representative 32-layer model uses about:

4 GiB

of BF16 KV cache for one 32K sequence.

Four independent requests at roughly the same length can push the raw cache requirement toward:

16 GiB

before model weights and other GPU allocations enter the picture.

The exact total in a serving engine depends on the active sequence lengths, block allocation, prefix sharing, cache format, and scheduler.

The principle remains:

context length tells you how large one request can become. Concurrency tells you how many of those requests must coexist.

That is why sizing an inference server from a single interactive test is risky.

Your model may work beautifully with one user and run out of cache capacity when ten users arrive.

Maximum context and useful context are different

Meta lists 128K context for Llama 3.1 and Llama 3.3. Some newer model families advertise substantially larger windows.

That number answers:

What sequence length was the model designed to support?

It does not answer:

What sequence length should I configure on this GPU?

The second question depends on:

model weights
+
KV cache
+
runtime memory
+
concurrency
+
latency requirements
+
your actual prompt distribution

This is why we deliberately started models such as Qwen3.6-27B with a smaller context limit in our deployment guide rather than automatically configuring the maximum the model supports.

A smaller limit can leave much more GPU memory available for real requests.

What is Grouped-Query Attention, and why does it matter?

Look back at the formula:

layers
× KV heads
× head dimension

The number of KV heads matters enormously.

In classic multi-head attention, each query head has its own key and value heads.

Grouped-Query Attention, or GQA, lets several query heads share a smaller number of key/value heads.

Conceptually:

32 query heads

instead of

32 K heads + 32 V heads

you might have

8 K heads + 8 V heads

That reduces the amount of K and V state that needs to be stored.

Meta explicitly uses GQA across Llama 3.1 and Llama 3.3 and describes the choice as improving inference scalability.

This is why you should use:

num_key_value_heads

rather than:

num_attention_heads

when calculating cache size for a GQA model.

Using the query-head count can overestimate the cache substantially.

MHA, GQA, and MQA change KV-cache size

At the same layer count and head dimension:

That does not make one architecture automatically better overall.

‍

‍

It shows why parameter count alone is insufficient for KV-cache sizing.

Two models with similar parameter counts and context windows can have very different cache requirements.

Look at the model configuration.

How to calculate your own model's KV cache

For many Hugging Face transformer models, the useful configuration fields are:

num_hidden_layers
num_key_value_heads
head_dim

If head_dim is not stored explicitly, it is often derived as:

hidden_size / num_attention_heads

Then use:

cache bytes per token =
num_hidden_layers
× 2
× num_key_value_heads
× head_dim
× bytes_per_element

Here is a simple calculator:

def kv_cache_gib(
   layers,
   kv_heads,
   head_dim,
   tokens,
   bytes_per_element=2,
   sequences=1,
):
   total_bytes = (
       layers
       * 2
       * kv_heads
       * head_dim
       * tokens
       * bytes_per_element
       * sequences
   )

   return total_bytes / (1024 ** 3)


print(
   kv_cache_gib(
       layers=32,
       kv_heads=8,
       head_dim=128,
       tokens=32768,
       bytes_per_element=2,
   )
)

Result:

4.0

So this configuration uses roughly:

4 GiB

of raw BF16 KV-cache storage for one 32K sequence.

Do not treat the output as a complete GPU-memory prediction.

It is the cache component of the memory budget.

Why model quantization does not solve KV-cache memory automatically

Suppose you quantize a model from BF16 to 4-bit.

The model weights become much smaller.

Great.

But if the serving engine still stores K and V tensors in BF16, the KV-cache calculation has not shrunk.

You might therefore go from:

large BF16 model
+
BF16 cache

to:

much smaller 4-bit model
+
the same BF16 cache

At short contexts, model-weight memory may still dominate.

At very long contexts or high concurrency, the cache can become an increasingly important part of the total.

This is why weight quantization and cache quantization need to be discussed separately.

The upcoming NVFP4 guide will focus primarily on model weights and compute. This article is about the growing inference state around them.

FP8 KV cache can roughly halve raw cache storage

vLLM currently supports several KV-cache data types, including BF16, FP16, and FP8 formats. Its auto mode uses the model's data type, while an explicit FP8 cache can reduce the number of bytes used by each cached value.

For the representative example above:

32K BF16 cache ≈ 4 GiB

becomes roughly:

32K FP8 cache ≈ 2 GiB

And:

128K BF16 cache ≈ 16 GiB

becomes roughly:

128K FP8 cache ≈ 8 GiB

That is a large difference.

It does not mean FP8 cache is universally free of quality tradeoffs. Quantizing cached attention state changes numerical precision, and you should evaluate the model and workload you intend to serve.

Memory savings are valuable because they buy something concrete:

more context
or
more concurrent requests
or
more room for the model

A vLLM example

Suppose a model supports far more context than your product needs.

You might deliberately serve it at 32K with FP8 KV cache:

vllm serve <model> \
 --max-model-len 32768 \
 --kv-cache-dtype fp8

Current vLLM versions expose the KV-cache dtype independently and also allow direct control of the memory budget allocated to the cache.

The point is not that these two flags are the universal optimal configuration.

The point is that context and cache precision are serving decisions.

You do not have to accept every maximum from the model card as your runtime configuration.

Our vLLM setup guide covers the rest of the serving environment.

What is PagedAttention?

There is another memory problem hiding inside KV cache.

Even if you know exactly how much information needs to be stored, serving requests of different lengths can use GPU memory inefficiently.

One user might have:

2K tokens

another:

19K

another:

7K

and their sequences keep growing and completing at different times.

Allocating large contiguous chunks of memory for each request can waste capacity through fragmentation and over-reservation.

The original vLLM work introduced PagedAttention to address this.

Instead of requiring one contiguous cache allocation per sequence, vLLM divides KV-cache storage into blocks inspired by virtual-memory paging. The research behind vLLM reported near-zero waste from KV-cache fragmentation and more flexible sharing of cached state between requests.

That is a major serving improvement.

But there is a distinction worth preserving:

PagedAttention makes KV-cache memory easier to manage. It does not make the underlying K and V information for every token free.

A 100K sequence still contains far more cached attention state than a 5K sequence.

We'll go deeper into that in our planned PagedAttention and continuous batching guide.

PagedAttention solves fragmentation, not the per-token memory cost

Think of a hotel.

A normal cache allocator might reserve an entire long corridor because a guest could eventually need every room.

Paged allocation lets you assign rooms in smaller chunks as the guest actually needs them.

That wastes less space.

It does not change how many rooms a guest occupies once they have filled them.

The same principle applies here.

Paging improves:

allocation
reuse
sharing
fragmentation

The model architecture and cache precision still determine how much data an active token requires.

This is why both kinds of optimization matter.

What is prefix caching?

Some requests share the same beginning.

Imagine an assistant where every request starts with:

a 10K-token system prompt
+
the same documentation

If the server recomputes and stores that exact prefix independently for every request, it duplicates work.

Prefix caching lets a serving system recognize previously computed blocks and reuse them.

vLLM's current prefix-cache manager hashes prompt-token blocks, looks for already computed blocks, and lets new requests reference those cached blocks where applicable.

That can reduce repeated prefill work and duplicated cache storage for shared prefixes.

It does not mean arbitrary conversations now share one KV cache.

Once requests diverge, their unique tokens need their own state.

Prefix caching and KV caching are not the same thing

The terms can sound interchangeable.

They solve different layers of the problem.

KV caching means:

Do not recompute the K and V states
for tokens this active sequence
has already processed.

Prefix caching means:

If another request has exactly the same
eligible prefix, reuse the already-computed
cache blocks for that prefix too.

The first is fundamental to efficient autoregressive decoding.

The second is a serving optimization across requests.

This distinction becomes important when evaluating claims about prompt caching or shared context.

Can the KV cache be moved to CPU RAM?

Yes, in some inference implementations.

Hugging Face Transformers supports cache strategies that can offload cache state away from the GPU, trading GPU-memory pressure for additional data movement. Its current cache documentation includes both offloaded and quantized cache strategies.

This can be useful when VRAM is the hard limit.

The tradeoff is similar to model-weight offloading:

less GPU memory
↔
more transfer overhead

A configuration becoming technically possible does not mean it has the same latency or throughput as keeping the cache resident on the GPU.

Benchmark the resulting workload.

Why sliding-window attention changes the calculation

The simple formula assumes every layer stores K and V states for the full active sequence.

Some architectures do not.

With sliding-window attention, a layer only attends over a limited recent window. Once that layer's cache reaches the window size, it does not need to keep growing indefinitely.

Hugging Face's current cache documentation specifically notes that caches for sliding-window and chunked-attention layers stop growing once their configured attention window is reached.

This means:

128K model context

does not necessarily mean:

every attention layer
stores 128K tokens of KV state

Architecture matters again.

Hybrid models make simple KV calculators less exact

Modern models increasingly mix attention mechanisms.

One model can contain:

full attention layers
+
sliding-window layers
+
recurrent or state-space components

Those layers do not necessarily have identical cache behavior.

vLLM's current cache system explicitly supports models containing different attention and cache types rather than assuming every layer uses one uniform full-attention cache.

That is why the simple formula should be treated as:

a very good sizing tool for standard full-attention transformers

rather than:

a universal law for every new architecture.

If you are deploying an unusual model, inspect its configuration and serving-engine documentation.

The model card cannot tell you your concurrency

This is the key production distinction.

A model vendor can tell you:

maximum context = 128K

It cannot know whether your application serves:

one user

or:

100 simultaneous users

Suppose each real request averages only 8K active tokens.

That looks modest.

At significant concurrency, the total number of active cached tokens can still become enormous.

The serving problem is therefore less:

Can this GPU fit a 128K context?

and more:

How many active tokens can this deployment sustain at the latency we need?

That is a much better capacity-planning question.

More context can reduce concurrency

Suppose a GPU has enough free cache memory for approximately:

128K tokens of KV state

in a particular model and precision.

You could theoretically spend that capacity as:

1 request × 128K

or roughly:

4 requests × 32K

or:

16 requests × 8K

Real schedulers and workloads are more complicated than that simple division, but it exposes the tradeoff.

The same GPU-memory budget can be spent on longer individual contexts or more simultaneous contexts.

This is why blindly maximizing max_model_len can be a poor production decision.

Model weights and KV cache compete for the same scarce resource

GPU memory has several claimants:

‍

‍

You cannot allocate the same gigabyte twice.

This makes quantizing model weights useful for more than fitting a bigger model.

Reducing weight memory can leave additional GPU capacity for KV cache.

Likewise, reducing KV-cache precision can leave room for more requests without changing the model weights.

That interaction is why a good inference configuration has to consider the whole memory budget.

How should you size an LLM endpoint?

I would work in this order:

1. Measure model-weight memory. Do not estimate a quantized checkpoint from parameter count if the real checkpoint exists.
2. Calculate approximate KV memory per token. Use the model's layer count, KV heads, head dimension, and intended cache precision.
3. Choose realistic context, not advertised maximum context. Look at actual prompt and output distributions.
4. Estimate concurrent active tokens. Five 20K requests can matter more than one 100K request.
5. Leave memory for the runtime. A calculation that ends at 31.99GB on a 32GB GPU is not a deployment plan.
6. Benchmark the serving engine. Real allocation, scheduling, prefix reuse, attention architecture, and quantization can all change the practical result.

That sequence is more reliable than picking a model because its weight file fits.

What does this mean for a 32GB RTX 5090?

A 32GB GPU can be a very capable inference device.

Its limit is still 32GB.

If a quantized model occupies:

16GB

that does not mean you have another 16GB available purely for context.

The runtime needs some of that space too.

But it does mean the remaining memory can support a meaningful cache.

This is why a quantized 27B model can be an interesting single-GPU workload while a 70B model generally moves toward multi-GPU serving.

Our Qwen3.6-27B guide shows the first case.

Our Llama 3.3 70B guide shows the second.

And our upcoming tensor parallelism guide will explain what happens to the memory problem once those weights and caches are distributed across several GPUs.

A bigger context window is not automatically a better product

Long context is useful.

It lets you provide:

larger documents
more conversation history
more retrieved passages
longer codebases
more examples

But every additional token can carry costs in:

memory
prefill time
attention work
latency
serving capacity

If your application works well with 16K tokens, configuring 128K does not make it eight times more capable.

It may simply give the server a much more expensive worst case.

Design around what users actually need.

Retrieval can be better than stuffing everything into context

A large context window creates a temptation:

We have 128K tokens, so let's put everything in the prompt.

That is often poor system design.

If only five paragraphs from a 500-page knowledge base are relevant to the question, retrieving those passages can be more efficient than repeatedly prefilling huge amounts of irrelevant text.

That is one reason RAG with Hivenet and long-context inference solve related but different problems.

Long context gives the model room.

Retrieval decides what deserves to occupy it.

The two approaches can work together.

The useful way to think about KV cache

The KV cache is the model's working memory for attention during generation.

It gives you speed because previous attention states do not have to be recreated every time another token appears.

But those states have to live somewhere.

So every serving decision becomes a tradeoff:

longer context
↔
more cache memory

higher concurrency
↔
more cache memory

higher cache precision
↔
more cache memory

more KV heads
↔
more cache memory

And the reverse:

GQA
FP8 cache
prefix reuse
paged allocation
sliding attention

can make that memory easier to use.

The important thing is to stop treating context length as a number that belongs only to the model.

Once the model is running, context length is an infrastructure decision too.

KV cache FAQ

What is KV cache in an LLM?

A KV cache stores the key and value tensors produced by attention layers for tokens the model has already processed. During autoregressive generation, the model reuses those tensors rather than recalculating them for every new token.

What does KV stand for?

KV stands for key-value, referring to the key and value vectors used by transformer attention.

Why does an LLM need a KV cache?

Without caching, the model would repeatedly recompute key and value states for previous tokens as it generated each new token. Caching trades additional memory for substantially less repeated computation during inference.

Is KV cache stored in VRAM?

For GPU-resident inference, KV cache is normally kept in GPU memory for fast access. Some frameworks also support cache offloading or other memory strategies when VRAM is limited.

Does KV cache grow with context length?

Yes, for standard full-attention dynamic caches. Each additional active token adds key and value state across the model's relevant attention layers. Sliding-window and other specialized architectures can change that behavior.

How do you calculate KV-cache size?

For a conventional full-attention transformer, a useful approximation is:

tokens
× layers
× 2
× KV heads
× head dimension
× bytes per element

Multiply by concurrent sequences when estimating several independent sequences of the same length.

Why is there a 2 in the KV-cache formula?

Because each cached token stores both a key tensor and a value tensor at every relevant attention layer.

Does model parameter count determine KV-cache size?

No. KV-cache size depends more directly on attention architecture, layer count, KV-head count, head dimension, sequence length, cache precision, and concurrency.

Does quantizing an LLM to 4-bit also make the KV cache 4-bit?

Not automatically. Model-weight precision and KV-cache precision are separate settings. A 4-bit model can still use a BF16 or FP16 KV cache.

Can vLLM use an FP8 KV cache?

Yes. Current vLLM versions support FP8 KV-cache formats in addition to higher-precision formats.

How much memory does FP8 KV cache save?

Compared with BF16 or FP16, FP8 uses roughly half the raw bytes per cached value. Actual total GPU-memory savings depend on the model and the rest of the serving stack.

Does PagedAttention reduce KV-cache size?

It primarily improves how cache memory is allocated and shared, reducing fragmentation and waste. It does not remove the fundamental per-token key/value state required by the attention architecture.

What is prefix caching?

Prefix caching lets a serving engine reuse previously computed KV-cache blocks when another request begins with the same eligible token prefix. This can avoid duplicated prefill computation and cache state.

Does a 128K context mean I should configure 128K?

No. It means the model supports sequences up to that limit under its model specification. The useful runtime limit depends on VRAM, KV-cache size, concurrency, latency, and your application's real context requirements.

Do prompt tokens use KV cache?

Yes. Prompt tokens are processed during prefill, and their key/value states are then used during subsequent decoding.

Do generated tokens use KV cache?

Yes. Each generated token adds its own key and value states to the active cache for subsequent decoding steps.

Why can a model load successfully and then run out of memory?

Loading proves that its weights and initial runtime allocations fit. Longer contexts or additional concurrent requests can then consume more KV-cache memory until the GPU no longer has enough capacity.

Does GQA reduce KV-cache memory?

Yes. Grouped-Query Attention uses fewer key/value heads than query heads, reducing how much key/value state must be stored. Meta uses GQA in Llama 3.1 and Llama 3.3 specifically to improve inference scalability.

‍