CUDA out of memory in PyTorch: diagnose and fix the cause

A CUDA out of memory error means a GPU allocation could not be satisfied. In PyTorch, start by checking the failing phase, the device being used, and the difference between live tensor memory and allocator reservations. Then reduce the workload or remove unnecessary references. Calling empty_cache() alone does not release tensors your program still needs.

This guide is for developers debugging PyTorch training or inference on an NVIDIA GPU. It provides a short diagnostic sequence, small inspection examples, and a way to verify each change. It does not assume that every memory failure is fragmentation or that buying a larger GPU is the first useful step.

Record the failure before changing settings

Save the full exception, the failed allocation size, GPU model, PyTorch version, input dimensions, batch size, and the point where execution stopped. Was it loading weights, running a forward pass, computing gradients, taking an optimizer step, or handling a larger request? Keep one failing input so that successive runs remain comparable.

Use this initial order:

1. Confirm which GPU the process sees and whether another workload occupies it.
2. Measure memory inside the failing process.
3. Reduce one workload dimension and repeat the same case.
4. Check for tensors retained across iterations.
5. Investigate allocator behavior only when the measurements support it.

An unavailable GPU, a missing kernel, or a driver compatibility error requires a different investigation. For an image-generation environment, our Stable Diffusion CUDA version guide helps separate software compatibility from memory pressure.

Check the device and other processes

The following commands inspect GPU memory and show the usual device/process summary. They do not stop jobs or change driver settings:

nvidia-smi --query-gpu=index,uuid,name,memory.total,memory.used,memory.free --format=csv
nvidia-smi

Compare the GPU UUID with the device assigned to your job. Containers and visibility settings can change the logical numbering your application sees. If a device is shared, identify the owner of a competing process before asking them to release it. Do not kill an unfamiliar process to make a diagnostic run fit.

NVIDIA's nvidia-smi reference describes the query fields and environment-specific limitations. Unsupported values can appear as N/A. Treat the output as a device-level observation, not a complete explanation of PyTorch's internal allocations.

Measure allocated, reserved, and free memory

These measurements answer different questions:

• memory_allocated() measures GPU memory occupied by this process's PyTorch tensors.
• memory_reserved() measures memory managed by its caching allocator. It includes allocated tensor memory; do not add the two values.
• mem_get_info() reports global device free and total memory through CUDA. Other processes can affect it.
• max_memory_allocated() records the tensor allocation peak since the statistics were last reset, or since the program began.

Insert this helper into the process you are investigating. Running it in a new Python process will not inspect another process's PyTorch allocator:

import torch

def report_cuda_memory(label, device=None):
    if not torch.cuda.is_available():
        print(f"{label}: CUDA is unavailable")
        return None
    if device is None:
        device = torch.cuda.current_device()
    gib = 1024 ** 3
    free, total = torch.cuda.mem_get_info(device)
    values = {
        "allocated_GiB": torch.cuda.memory_allocated(device) / gib,
        "reserved_GiB": torch.cuda.memory_reserved(device) / gib,
        "peak_allocated_GiB": torch.cuda.max_memory_allocated(device) / gib,
        "device_free_GiB": free / gib,
        "device_total_GiB": total / gib,
    }
    print(label, values)
    return values

# Call inside your existing training or inference process.
report_cuda_memory("before workload")

Call it at consistent boundaries: before the batch, after the forward pass, after backward, and after the optimizer step. Compare several iterations, including the largest expected input. The calls are separate observations, not an atomic snapshot; another process can allocate memory between them. Keep the labels and results with the failing configuration.

Match the first change to the observed pattern

• The first representative batch fails: try a smaller microbatch or input. Record whether the failing phase changes.
• Allocated memory grows across otherwise similar iterations: inspect lists, callbacks, metrics, and references to previous outputs.
• Only unusually large inputs fail: record their dimensions and test an explicit size limit or a separate processing path.
• Device usage is high while this process reports little tensor memory: examine other processes and allocations outside PyTorch.

These are starting hypotheses. A smaller batch completing once is useful evidence, but it does not establish that the longest training run or busiest inference period will fit. Use the same input order when comparing changes so that an easier batch does not look like a fix.

Remove accidental tensor retention

Metrics and saved outputs can keep unnecessary computation history alive. For a scalar loss that you only want to log, store a Python number:

# loss is a scalar tensor from the existing training step.
# Keep the list outside the training loop.
loss_values.append(loss.detach().item())

PyTorch's memory troubleshooting FAQ explains this retention problem. Inspect object attributes and callbacks as well as local variables. Removing one reference does not help if another reference still holds the tensor. Keep diagnostic histories bounded too, even when their contents have moved to CPU memory.

Use inference settings for inference

Evaluation mode and gradient tracking are separate controls. For a model that returns one tensor, this small helper sets evaluation behavior and runs without autograd tracking:

import torch

def predict_to_cpu(model, batch):
    # model and batch must already be on the same device.
    # This example expects a single Tensor output.
    model.eval()
    with torch.inference_mode():
        return model(batch).cpu()

inference_mode() does not call model.eval() for you. Its tensors also have restrictions if later used in gradient-recorded computations. Use no_grad() where those restrictions do not fit. Adapt output handling for dictionaries or tuples, and return the model to training mode when resuming training.

For language-model serving, input length, output length, and simultaneous requests also need explicit limits. Our KV cache guide explains why fitting model weights is only part of inference capacity planning.

Reduce training memory with a deliberate trade-off

Microbatches and gradient accumulation

Smaller microbatches can reduce the work held at once. Gradient accumulation lets several microbatches contribute to one optimizer update. Follow the loss normalization and optimizer timing in PyTorch's accumulation examples, especially when combining accumulation with gradient scaling. Handle an incomplete final group explicitly. Treat changes to training behavior as something to validate, not just a memory setting.

Mixed precision

Automatic mixed precision selects lower precision for eligible operations while retaining higher precision where needed. Check the hardware, supported operators, and numerical behavior before enabling it. Do not promise that total memory will be cut in half. Compare loss behavior and output quality alongside the memory peak.

Activation checkpointing

Activation checkpointing recomputes selected forward operations during backward to reduce saved intermediates. It trades computation for memory. PyTorch recommends explicitly passing use_reentrant=False for the non-reentrant variant. Validate gradients and runtime with your model. This is different from writing a training checkpoint to disk so a job can resume later.

Understand what empty_cache() can release

empty_cache() releases unused cached blocks for other GPU applications. It does not free memory occupied by live tensors or enlarge the GPU. It can help with fragmentation in some cases, but it is not a substitute for reducing the working set or fixing retained references.

If you restart your own notebook kernel, first preserve the work you need and check whether the failure returns under the same input. A restart can give you a clean comparison; it does not identify the cause by itself.

Investigate fragmentation after measuring it

A large reserved-versus-allocated gap alone does not prove fragmentation. Inspect memory_summary() or memory_stats() before tuning the allocator. In the current CUDA memory management notes, PYTORCH_ALLOC_CONF is the configuration variable; PYTORCH_CUDA_ALLOC_CONF remains a compatibility alias. Options depend on the allocator backend. For example, max_split_size_mb is a last-resort native-allocator setting and is ignored by cudaMallocAsync.

If the short report is insufficient, follow PyTorch's memory snapshot workflow on a small reproduction. It can reveal allocation history, but the profiler does not see all allocations made outside PyTorch, such as some communication-library memory. Review captured paths and tracebacks before sharing a snapshot.

Verify the fix before choosing more hardware

Repeat the original failing case, then test the largest supported input and a sustained run. Record the memory peak, completion, throughput, and relevant quality checks. Change one variable at a time. Keep a successful configuration with package versions so that the next deployment does not quietly undo it.

If the workload still needs more memory after reasonable reductions, compare a larger-memory device with a supported sharding or offload approach. Ordinary DistributedDataParallel uses model replicas; adding a second GPU does not automatically make its memory available to one replica. For A100-specific setup and measurement, see our PyTorch on A100 guide.

Before renting a larger instance, define the exact workload you need to finish and the memory measurements you will compare. A successful small example establishes a starting point. Your real acceptance test should include the input sizes, concurrency, and runtime that caused the original failure.