Cross-Entropy Loss
Cross-entropy loss is a loss function that measures the difference between a predicted probability distribution and the true (target) distribution, commonly used to train classification models.
Definition
Cross-entropy loss is a loss function that measures the difference between a predicted probability distribution and the true (target) distribution, commonly used to train classification models.
Overview
Cross-entropy loss is derived from information theory, where cross-entropy measures the average number of bits needed to represent events drawn from a true distribution when using a code optimized for a different, predicted distribution. In machine learning, this translates into a loss function that penalizes a model heavily when it assigns low probability to the correct class, and lightly when it assigns high probability to the correct class — making it a natural fit for training probabilistic classifiers. For binary classification, the loss is typically called binary cross-entropy (or log loss), applied to the output of a sigmoid activation. For multi-class classification, categorical cross-entropy is applied to the output of a softmax function, which converts raw model outputs (logits) into a probability distribution over classes; in this setting the loss simplifies to the negative log of the predicted probability assigned to the correct class. Because of this logarithmic penalty, cross-entropy loss produces especially large gradients when the model is confidently wrong, which drives fast correction during training — a key reason it outperforms alternatives like squared error for classification tasks with probabilistic outputs. Cross-entropy loss (often combined into a single "softmax cross-entropy" or "log-softmax + negative log-likelihood" operation for numerical stability) is the standard training objective for image classifiers, and is the direct ancestor of the "next-token prediction" loss used to pretrain large language models, where the model is trained to assign high probability to the actual next token in a sequence across a huge vocabulary. Label smoothing, a common regularization technique, modifies the target distribution slightly (assigning a small probability mass to incorrect classes) to prevent the model from becoming overconfident.
Key Concepts
- Measures the divergence between predicted and true probability distributions
- Binary cross-entropy applies to sigmoid outputs for two-class problems
- Categorical cross-entropy applies to softmax outputs for multi-class problems
- Penalizes confident wrong predictions much more heavily than uncertain ones
- Produces large gradients when predictions are confidently incorrect, speeding correction
- Standard loss function for image classification and language model pretraining
- Often numerically combined with log-softmax for stable computation
- Compatible with label smoothing to prevent overconfident predictions
Use Cases
Frequently Asked Questions
From the Blog
What Is a Loss Function in Machine Learning
A loss function is the formula that measures how wrong a model's predictions are, giving training a single number to minimize so the model can improve.
Read More AI & TechnologyWhat Is Cross-Validation in Machine Learning
Cross-validation tests a model on multiple data splits instead of one, giving a reliable estimate of how it will perform on unseen data. Here is how it works.
Read More AI & TechnologyFine-Tuning Loss Won't Drop: A Debugging Checklist
A fine-tuning loss curve that refuses to move almost always means the gradients are not reaching the weights you think they are. Work through label masking, tokenizer and template mismatch, learning rate, and which parameters actually have requires_grad set — in that order.
Read More AI & TechnologyReranking in RAG: When a Cross-Encoder Earns Its Latency
A cross-encoder reranker earns its latency when your first-stage retriever has high recall at a wide k but poor ordering in the top few. This article explains the two-stage pattern, the recall condition that makes reranking worthwhile, and how to measure whether it is paying for itself.
Read More