What are Loss Functions in Machine Learning?
Understand loss functions in machine learning: what they measure, how MSE and cross-entropy differ, and how they drive gradient descent and model training.
Expected Interview Answer
A loss function measures how far a model's predictions are from the true targets, producing a single number the training process tries to minimise.
During training, the optimiser adjusts model parameters to reduce the loss via gradient descent, so the loss defines what "good" means for the model. Different tasks use different losses: mean squared error and mean absolute error for regression, binary or categorical cross-entropy for classification, and specialised losses like hinge or Huber for particular cases. The choice matters because it shapes the gradients, penalises certain errors more than others, and directly influences what the model learns to prioritise.
- Quantifies prediction error as a single optimisable number
- Provides the gradient signal that drives learning
- Lets you tailor penalties to the task (regression vs classification)
- Controls robustness to outliers via the chosen formula
- Enables comparison between models on the same objective
AI Mentor Explanation
Think of a bowler aiming at the stumps. The loss function is the distance the ball misses by: a delivery brushing the off stump scores a tiny error, one sprayed down the leg side a huge one. Coaching drills work to shrink that miss distance over many balls, exactly as training minimises loss, nudging the bowler's action until deliveries land where intended.
Step-by-Step Explanation
Step 1
Make a prediction
The model produces an output for a batch of inputs given its current parameters.
Step 2
Compare to targets
The loss function measures the discrepancy between predictions and the true labels.
Step 3
Aggregate the error
Per-example errors are combined (usually averaged) into a single scalar loss value.
Step 4
Compute gradients
Backpropagation differentiates the loss with respect to each parameter.
Step 5
Update parameters
The optimiser steps parameters in the direction that reduces the loss, and the cycle repeats.
What Interviewer Expects
- Clear definition of loss as an error measure to minimise
- Correct pairing of losses with tasks (MSE for regression, cross-entropy for classification)
- Understanding of how loss connects to gradients and optimisation
- Awareness of outlier robustness (MAE/Huber vs MSE)
- Difference between loss (per-batch) and cost/objective (overall)
Common Mistakes
- Confusing loss functions with evaluation metrics like accuracy
- Using MSE for classification instead of cross-entropy
- Ignoring how MSE amplifies outliers versus MAE
- Thinking a lower training loss always means a better model (overfitting)
- Forgetting the loss must be differentiable for gradient descent
Best Answer (HR Friendly)
“A loss function is a scorecard that tells the model how wrong its predictions are. Training works by repeatedly adjusting the model to make that score as low as possible, so a good loss function is what teaches the model to improve.”
Code Example
import numpy as np
from sklearn.metrics import mean_squared_error, log_loss
# Regression: mean squared error
y_true = np.array([3.0, -0.5, 2.0, 7.0])
y_pred = np.array([2.5, 0.0, 2.0, 8.0])
mse_manual = np.mean((y_true - y_pred) ** 2)
print(mse_manual, mean_squared_error(y_true, y_pred))
# Binary classification: cross-entropy (log loss)
y_cls = np.array([1, 0, 1, 1])
p_pred = np.array([0.9, 0.2, 0.7, 0.6])
print(log_loss(y_cls, p_pred))from tensorflow import keras
# Regression head -> MSE
reg = keras.Sequential([keras.layers.Dense(1)])
reg.compile(optimizer='adam', loss='mse')
# Multi-class head -> categorical cross-entropy
clf = keras.Sequential([keras.layers.Dense(3, activation='softmax')])
clf.compile(optimizer='adam', loss='categorical_crossentropy')Follow-up Questions
- Why is cross-entropy preferred over MSE for classification?
- How does MSE differ from MAE in handling outliers?
- What is the Huber loss and when would you use it?
- What is the difference between a loss function and an evaluation metric?
- Why must a loss function generally be differentiable?
MCQ Practice
1. Which loss function is most appropriate for a regression problem?
Mean squared error measures the squared difference between predicted and true continuous values, standard for regression.
2. Compared with MAE, mean squared error is more sensitive to:
Squaring the error means large deviations dominate the loss, so MSE penalises outliers heavily.
3. Why must most loss functions be differentiable?
Gradient-based optimisers need the derivative of the loss with respect to parameters to know how to update them.
Flash Cards
What is a loss function? — A measure of how far predictions are from targets, expressed as a single number the optimiser minimises.
Regression vs classification loss? — Regression typically uses MSE or MAE; classification uses binary or categorical cross-entropy.
MSE vs MAE on outliers? — MSE squares errors so it penalises outliers heavily; MAE treats errors linearly and is more robust.
Loss vs metric? — Loss is the differentiable objective optimised during training; a metric like accuracy is for human evaluation and need not be differentiable.