PyTorch eval, no_grad, and inference_mode: which to use

In PyTorch, model.eval() changes how the model’s layers behave. torch.no_grad() and torch.inference_mode() control gradient-related work. Ordinary prediction usually needs both decisions: evaluation behavior for the model and an appropriate gradient mode for the forward pass.

The distinction matters when validation changes BatchNorm statistics, predictions retain a computation graph, or frozen features later enter a trainable layer. The examples below were executed on CPU with PyTorch 2.15.0. They demonstrate correctness, not GPU speed or memory savings.

Choose the combination for the task

Task	Model behavior	Gradient mode
Train model parameters	Usually model.train()	Normal gradient recording
Predict without any later backward pass	model.eval()	inference_mode() is a suitable starting point
Validate without gradients	model.eval()	no_grad(), or inference_mode() if its tensor restrictions fit
Extract frozen features for a trainable head	backbone.eval(); head.train()	no_grad() around the backbone only
Compute gradients with respect to an input	Often model.eval()	Normal gradient recording; input requires gradients

PyTorch’s autograd notes distinguish module evaluation from gradient modes. Treat them as separate controls. A model can be in evaluation mode while autograd records its operations.

What model.eval() changes

Module.eval() is equivalent to train(False). It changes the training flag of the module and its child modules. Layers that use that flag can then switch behavior. It does not disable gradients or set parameter requires_grad flags to false.

Dropout randomly masks values during training and acts as an identity in evaluation mode. BatchNorm normally updates running statistics during training and uses them during evaluation. With track_running_stats=False, it uses batch statistics in both modes.

We checked both behaviors locally: a training Dropout layer still masked values inside no_grad(), and training BatchNorm still updated its running mean. Calling eval() changed those behaviors in the default configurations. Suppressing gradients alone therefore does not prepare these layers for ordinary validation.

A small prediction example

Run this as a standalone Python script in an environment with PyTorch installed. The model has random weights and is not trained; the example shows execution state rather than useful predictions.

import torch
from torch import nn

torch.manual_seed(7)
model = nn.Sequential(
    nn.Linear(4, 8),
    nn.ReLU(),
    nn.Dropout(0.5),
    nn.Linear(8, 2),
)
inputs = torch.randn(3, 4)

model.eval()
with torch.inference_mode():
    logits = model(inputs)

print(logits.shape, logits.requires_grad, torch.is_inference(logits))

The observed output was torch.Size([3, 2]) False True: a three-row output, no gradient requirement, and an inference tensor. In a real application, load the trained weights first and place the model and inputs on compatible devices and dtypes. Our PyTorch save-and-load guide covers checkpoint handling.

inference_mode() skips gradient recording and additional autograd bookkeeping, including view tracking and version-counter updates. It does not call eval(). Tensors created inside it also have restrictions when used later by autograd.

See the mode differences directly

Append this comparison to the prediction script. Its parameters still require gradients, so the evaluation-only row records a graph even though the input does not request one:

from contextlib import nullcontext

cases = [
    ("train + no_grad", True, torch.no_grad),
    ("eval + default grad", False, nullcontext),
    ("eval + no_grad", False, torch.no_grad),
    ("eval + inference_mode", False, torch.inference_mode),
]

for label, training, context in cases:
    model.train(training)
    with context():
        output = model(inputs)
    print(label, model.training, output.requires_grad,
          torch.is_inference(output))

Case	model.training	output.requires_grad	Inference tensor
train + no_grad	True	False	False
eval + default grad	False	True	False
eval + no_grad	False	False	False
eval + inference_mode	False	False	True

All four rows matched in the CPU test. The evaluated model also produced matching outputs under no_grad() and inference_mode(). This does not guarantee identical results for every model, device, or operation, and it is not a performance comparison.

When no_grad() is the better fit

no_grad() suppresses reverse-mode gradient recording within its scope while producing ordinary tensors. It is useful when a result will later be an input to a trainable computation.

This standalone example freezes a feature extractor, then computes gradients for a separate classification head:

import torch
from torch import nn
from torch.nn import functional as F

torch.manual_seed(7)
backbone = nn.Sequential(nn.Linear(4, 8), nn.ReLU())
head = nn.Linear(8, 2)
backbone.requires_grad_(False)
backbone.eval()
head.train()

inputs = torch.randn(3, 4)
targets = torch.tensor([0, 1, 0])
with torch.no_grad():
    features = backbone(inputs)

loss = F.cross_entropy(head(features), targets)
loss.backward()

print(head.weight.grad is not None)
print(all(p.grad is None for p in backbone.parameters()))

The two printed checks both returned True. The head received gradients, and the fresh backbone parameters did not. This performs one backward pass, not an optimizer update. An actual training loop also needs gradient clearing and an optimizer step; our gradient accumulation guide shows those responsibilities.

Replacing only the feature-extraction context with inference_mode() caused the tested linear head to reject the resulting tensor because it needed to save that input for backward. The input itself does not need gradients for the head to need its values when computing weight gradients.

If inference tensors must cross that boundary, cloning them outside inference mode can create ordinary tensors. We checked that workaround on this example. It adds a copy, so choosing no_grad() at the feature-extraction boundary is often clearer. Cloning does not recover a gradient path through computations that were never recorded.

Keep gradients enabled when analyzing an input

Some evaluation tasks need derivatives, such as examining how a score changes with its input. Evaluation mode can still be appropriate, but wrapping the forward pass in a gradient-disabling context would remove the path you need.

import torch
from torch import nn

torch.manual_seed(7)
model = nn.Linear(4, 2)
model.requires_grad_(False)
model.eval()
inputs = torch.randn(1, 4, requires_grad=True)

score = model(inputs)[0, 0]
(input_gradient,) = torch.autograd.grad(score, inputs)
print(input_gradient.shape)

This example returned torch.Size([1, 4]). We also checked that the gradient equals the first row of the linear layer’s weight matrix. The model’s parameters are frozen, yet the input can still receive a derivative. torch.autograd.grad returns that derivative directly.

Restore training behavior deliberately

After validation in a training loop, call model.train() before the next training forward pass. Leaving a no-grad or inference context restores the previous context state; it does not reverse a call to model.eval().

For a frozen backbone with a trainable head, set the intended submodule states explicitly. Calling train() on a parent also changes its children, which can undo a deliberate backbone evaluation setting. A top-level flag alone cannot describe every mixed-mode model.

The gradient contexts are thread-local. Put the appropriate context around execution in each worker that needs it. The model’s training flag is model state, so a shared serving model should not be switched between train and eval by concurrent prediction requests. Configure its intended state during setup.

Common mistakes to check

• Validation changes running statistics: check module training flags, especially BatchNorm, rather than only the gradient context.
• Prediction outputs retain graphs: inspect whether gradients are enabled and whether lists or caches keep those outputs alive.
• A trainable head rejects inference features: use a no-grad feature boundary or create an ordinary copy outside inference mode.
• Input-gradient analysis fails: check the input’s gradient flag and whether the forward pass ran with gradient recording enabled.
• Training behaves differently after validation: restore the intended model and submodule states before resuming.

detach() disconnects a tensor from its graph and shares its storage. Detaching an output after a forward pass does not prevent the graph-building work that already happened. If inference memory continues to grow, our CUDA out-of-memory guide covers retained outputs and other causes.

Two details matter for specialized code: no_grad() has an exception for factory functions with an explicit gradient option, and it does not disable forward-mode automatic differentiation. Inference mode disables forward-mode AD as well. Benchmark the chosen mode on the actual workload before claiming a speed or memory improvement; the small CPU checks here establish behavior only.