What is a Loss Function in Machine Learning?
Learn what a loss function is, how MSE and cross-entropy work, the difference between loss and cost, and how it drives gradient descent training.
Expected Interview Answer
A loss function is a mathematical function that measures how far a model's predictions are from the true target values for a single example, producing a number that training algorithms like gradient descent try to minimize.
The choice of loss function depends on the task: mean squared error (MSE) is standard for regression, penalizing larger errors quadratically, while cross-entropy loss (log loss) is standard for classification, penalizing confident wrong predictions heavily and rewarding confident correct ones. The loss is computed per example, and the overall objective during training is typically the average loss across a batch, called the cost function, though the terms loss and cost are often used interchangeably in practice. During training, the model's parameters are updated via gradient descent using the gradient of this loss, so the choice of loss function directly shapes what 'better' means for the model and what kind of errors it is penalized most for making. Choosing the wrong loss function for a task, like using MSE for a classification problem, produces poorly calibrated probabilities and slower, less stable convergence.
- Precisely defines what 'good' predictions mean for a given task
- Directly drives parameter updates via gradient descent
- Different loss functions let you weight error types differently (e.g. Huber loss is robust to outliers)
- Enables comparing models objectively on the same numeric scale
- Choosing task-appropriate loss (MSE vs cross-entropy) improves calibration and convergence
AI Mentor Explanation
A loss function is like a coach's scoring rubric for a net session, assigning a penalty number to each mistimed shot based on how far off the technique was. A rubric that heavily penalizes wild swings and mildly penalizes small flaws shapes practice differently than one treating every mistake the same, just like choosing MSE versus cross-entropy shapes training.
Step-by-Step Explanation
Step 1
Identify the task type
Determine whether you're solving regression, binary classification, or multi-class classification, since this determines the appropriate loss family.
Step 2
Choose the loss function
Use MSE or MAE for regression, binary cross-entropy for binary classification, or categorical cross-entropy for multi-class classification.
Step 3
Compute the loss for a batch
For each example in a training batch, compute the loss between prediction and true label, then average to get the batch cost.
Step 4
Compute gradients of the loss
Use backpropagation (for neural nets) or direct differentiation to compute the gradient of the loss with respect to model parameters.
Step 5
Update parameters via gradient descent
Use the gradients to update weights in the direction that reduces the loss, repeating over many batches and epochs.
Step 6
Monitor the loss curve
Track training and validation loss over time to check for convergence, overfitting, or the need for a different loss or learning rate.
What Interviewer Expects
- Defines a loss function as measuring prediction error for a single example
- Names task-appropriate losses (MSE for regression, cross-entropy for classification)
- Distinguishes loss (per example) from cost (average over a batch)
- Connects the loss function to gradient descent and parameter updates
- Can discuss a robust loss like Huber loss and why it helps with outliers
Common Mistakes
- Using MSE for a classification problem instead of cross-entropy
- Confusing loss function with evaluation metric (e.g. accuracy is not differentiable and not used as a training loss)
- Not knowing why cross-entropy heavily penalizes confident wrong predictions
- Treating loss and cost as entirely different concepts rather than per-example vs batch-averaged
- Forgetting that the loss function must be differentiable for gradient-based optimization
Best Answer (HR Friendly)
“A loss function is a formula that scores how wrong a model's prediction was for a given example, giving a bigger penalty for bigger mistakes. During training, the model repeatedly adjusts itself to make this penalty as small as possible, which is how it gradually gets better at its task.”
Code Example
import torch
import torch.nn as nn
# Regression: Mean Squared Error
mse_loss = nn.MSELoss()
pred_reg = torch.tensor([3.2, 5.1])
target_reg = torch.tensor([3.0, 5.0])
print("MSE:", mse_loss(pred_reg, target_reg).item())
# Binary classification: Binary Cross-Entropy
bce_loss = nn.BCELoss()
pred_cls = torch.tensor([0.9, 0.2]) # predicted probabilities
target_cls = torch.tensor([1.0, 0.0]) # true labels
print("BCE:", bce_loss(pred_cls, target_cls).item())Follow-up Questions
- Why is cross-entropy loss preferred over MSE for classification tasks?
- What is the difference between a loss function and an evaluation metric?
- What is Huber loss and when is it preferred over MSE?
- How does the choice of loss function affect gradient descent convergence?
- What is focal loss and why is it used for imbalanced classification?
MCQ Practice
1. Which loss function is standard for binary classification tasks?
Binary cross-entropy (log loss) is the standard loss for binary classification, heavily penalizing confident wrong predictions.
2. What is the key difference between 'loss' and 'cost' as commonly used in ML?
Loss usually refers to the error on a single example, while cost (or objective) is the aggregated, typically averaged, loss over a batch or dataset.
3. Why can't accuracy typically be used directly as a training loss function?
Accuracy is a step-function metric with zero gradient almost everywhere, so gradient descent cannot use it directly to update parameters.
Flash Cards
What does a loss function measure? — How far a model's prediction is from the true target value for a single example.
Which loss is standard for regression, and which for classification? — Mean squared error (MSE) for regression; cross-entropy (log loss) for classification.
What is the difference between loss and cost? — Loss is typically per-example error; cost is the averaged loss across a batch or dataset.
Why must a loss function be differentiable? — Gradient-based optimizers like gradient descent need gradients of the loss to update model parameters.