PyTorch CrossEntropyLoss: logits, targets, and reduction

PyTorch CrossEntropyLoss compares a model’s class scores with the expected targets. For ordinary multiclass classification, pass raw logits with shape [N, C] and integer class labels with shape [N]. Do not apply softmax to the logits first.

This guide uses small, deterministic CPU examples to show what the loss calculates, how target shapes affect it, and why weighting changes the denominator of a mean. The examples and 18 checks ran on Windows with Python 3.12.14 and PyTorch 2.15.0+cpu. They need no downloaded model or dataset.

Start with one class label per example

Suppose a model chooses among three mutually exclusive classes. Each row of its output contains three scores. The target identifies the correct column, starting at zero. A batch of two examples therefore uses logits shaped [2, 3] and targets shaped [2].

torch.nn.CrossEntropyLoss() is the module interface; torch.nn.functional.cross_entropy provides the functional interface used below. The functional API reference lists its arguments. Run these snippets in an environment with PyTorch installed, keeping the imports from the first block.

import torch
import torch.nn.functional as F

logits = torch.tensor([[2.0, 0.0, -1.0],
                       [0.0, 1.0, 3.0]], requires_grad=True)
targets = torch.tensor([0, 1], dtype=torch.long)

per_item = F.cross_entropy(logits, targets, reduction="none")
loss = per_item.mean()
loss.backward()

print("Per item:", [round(x, 6) for x in per_item.tolist()])
print("Mean:", round(loss.item(), 6))
print("Gradient shape:", tuple(logits.grad.shape))

The observed output was:

Per item: [0.169846, 2.169846]
Mean: 1.169846
Gradient shape: (2, 3)

The first row gives class 0 the highest score and also has target 0, so its loss is small. The second row favors class 2 but has target 1, producing the larger loss. Their mean is 1.169846. This number is a loss value, not an accuracy percentage.

Calling backward() fills the gradient tensor for the logits. In a model, that calculation continues through the operations that produced them. This example has no optimizer step and makes no claim about learning quality on a dataset.

Pass logits, not softmax probabilities

For class-index targets and default settings, the per-example calculation selects the target class from the log-softmax output and negates it. Our checks reproduced the result with F.nll_loss(F.log_softmax(logits, dim=1), targets). The LogSoftmax reference describes that transformation.

Applying softmax before CrossEntropyLoss changes its input scores. In this test, F.cross_entropy(logits.softmax(dim=1), targets) returned 1.022646 instead of 1.169846. The call succeeds, which makes the mistake easy to miss. Use probabilities when you need to inspect predictions, while keeping the original logits for this loss.

A shared offset to every score in a row does not change the relative class probabilities. Adding 10,000 to these logits preserved the loss within the test tolerance. Use the library’s stable calculation instead of implementing it through separate exponentiation and logarithm operations.

Check target shape and dtype before changing the model

For the class-index path in this example, use torch.long targets. Each value must identify a class from 0 through C - 1, apart from the configured ignored value. The CrossEntropyLoss reference describes both the index and probability-target forms.

• Floating-point labels shaped [N]: our runtime rejected them. Convert categorical indices to long only after checking that they actually represent integer class IDs.
• Labels shaped [N, 1]: the extra dimension was rejected for [N, C] logits. Fix the target construction; do not reshape tensors indiscriminately.
• A label equal to C: it is outside the valid class range. With three output columns, target 3 failed because valid IDs are 0, 1 and 2.

Inspect one batch before a long training run. Check its shapes, label mapping and finite values at the point where the loss is called. Our DataLoader guide covers how batches are assembled; a data-pipeline error can surface later as a loss error.

Choose the reduction deliberately

reduction="none" keeps one loss per example here. "sum" adds those losses, and the default "mean" averages them in the unweighted, non-ignored case. We verified all three against the first example.

Do not assume every mean is a sum divided by the batch size. For class-index targets with class weights, no label smoothing and a positive effective weight, the mean divides the summed weighted losses by the weights of the valid target classes. An ignored example contributes neither a loss nor a target weight.

logits = torch.tensor([[2.0, 0.0, -1.0],
                       [0.0, 1.0, 3.0],
                       [1.0, 2.0, 0.0]])
targets = torch.tensor([0, 1, -100], dtype=torch.long)
weights = torch.tensor([1.0, 3.0, 2.0])
valid = targets != -100

if not valid.any().item():
    raise ValueError("This batch has no supervised targets")
denominator = weights[targets[valid]].sum()
if denominator.item() <= 0:
    raise ValueError("The valid targets have zero total weight")

parts = F.cross_entropy(logits, targets, weight=weights,
                        ignore_index=-100, reduction="none")
manual_mean = parts.sum() / denominator
builtin_mean = F.cross_entropy(logits, targets, weight=weights,
                               ignore_index=-100)
torch.testing.assert_close(manual_mean, builtin_mean)
print("Weighted parts:", [round(x, 6) for x in parts.tolist()])
print("Denominator:", denominator.item())
print("Weighted mean:", round(builtin_mean.item(), 6))

This produced:

Weighted parts: [0.169846, 6.509538, 0.0]
Denominator: 4.0
Weighted mean: 1.669846

The valid targets are classes 0 and 1, whose weights are 1 and 3. The denominator is therefore 4, not two examples or three rows. Dividing the same weighted sum by two would produce 3.339692 and a different gradient scale. Class weights change how examples contribute; choose them for a stated modeling reason and evaluate the result.

Handle ignored and empty supervision explicitly

ignore_index=-100 skips positions carrying that target value. It does not remove an output class from the model’s softmax calculation. It applies to index targets; probability targets need their own masking and reduction logic.

In the tested runtime, a default mean over a batch where every target was ignored returned NaN. A batch whose valid targets all had zero class weight did the same. The weighted example checks both conditions before division. In a training loop, decide explicitly whether to reject such a batch or skip its update; replacing a nonfinite loss with an arbitrary zero can conceal a labeling problem.

The denominator also matters when combining microbatches. Two microbatches with different valid target weights should not automatically contribute equally to a reported mean. Our gradient accumulation guide covers update timing; preserve the intended loss normalization when adapting it to weighted or masked targets.

Use probability targets only when they match the task

Soft targets describe a distribution across the same mutually exclusive classes, for example when a label deliberately assigns partial mass to several classes. They are floating-point tensors with the same shape as the logits. Validate their values and their sum along the class dimension.

logits = torch.tensor([[2.0, 0.0, -1.0], [0.0, 1.0, 3.0]])
soft_targets = torch.tensor([[0.7, 0.2, 0.1], [0.1, 0.3, 0.6]])
if not torch.isfinite(soft_targets).all().item():
    raise ValueError("Targets must be finite")
if ((soft_targets < 0) | (soft_targets > 1)).any().item():
    raise ValueError("Target probabilities must be between 0 and 1")
torch.testing.assert_close(soft_targets.sum(dim=1), torch.ones(2))

loss = F.cross_entropy(logits, soft_targets)
manual = -(soft_targets * F.log_softmax(logits, dim=1)).sum(dim=1).mean()
torch.testing.assert_close(loss, manual)
print("Soft-target mean:", round(loss.item(), 6))

The soft-target mean was 0.969846. The check reconstructs it by multiplying target probabilities by log probabilities, summing across classes and averaging across examples. This is a different target form from the weighted index example; do not reuse that example’s weight-sum denominator without checking the applicable formula.

Our test also passed an invalid target tensor filled with 2.0. PyTorch returned a finite loss rather than rejecting the distribution. A successful call therefore does not prove that soft labels are valid. The explicit checks above catch this kind of input mistake before the loss is calculated.

Keep the class dimension in the right place

Dense prediction adds spatial dimensions. For image segmentation logits shaped [N, C, H, W], class-index targets use [N, H, W]. Keeping the reduction disabled produces a loss at each target position.

logits = torch.arange(24, dtype=torch.float32).reshape(2, 3, 2, 2) / 10
targets = torch.tensor([[[0, 1], [2, 0]], [[2, 1], [0, 2]]])
pixel_losses = F.cross_entropy(logits, targets, reduction="none")
print("Pixel loss shape:", tuple(pixel_losses.shape))

The output shape is (2, 2, 2). We verified these eight pixel losses against an equivalent [8, 3] classification calculation after moving the class dimension to the end and flattening. Simply flattening the original logits without that dimension move would mix the class scores incorrectly.

Separate multiclass from independent labels

When an example can have several independent labels at once, forcing those labels into one class distribution usually expresses the wrong task. BCEWithLogitsLoss combines a sigmoid-based binary loss with logits and supports independent binary targets. Binary classification can also use two logits with CrossEntropyLoss; match the output and target convention throughout the model.

Before scaling up, confirm one batch’s class axis, target representation, per-item losses and reduction denominator. These CPU checks establish the behavior of the illustrated calculations, including backward and error cases. They do not establish GPU throughput, convergence, calibration or accuracy on a real dataset. The environment emitted a NumPy initialization warning because NumPy was absent; these tensor-only examples completed without it.