Training with Backpropagation
Backpropagation is the algorithm that makes training deep neural networks computationally feasible. It answers a precise question: for a given loss value, how much does each individual weight in the network need to change to reduce that loss? Rather than perturbing each weight one at a time and re-measuring the loss (which would be prohibitively slow for millions of parameters), backpropagation applies the chain rule of calculus to compute all the gradients in a single backward pass through the network, reusing intermediate computations along the way. Combined with gradient descent, which uses those gradients to update the weights, backpropagation forms the training loop that underlies virtually every modern neural network.
Cricket analogy: Instead of a coach adjusting a bowler's grip one finger at a time and rebowling an entire over to see what changes, backpropagation calculates in one pass exactly how each finger position should shift to reduce no-balls.
Forward Pass and Loss
Training begins with a forward pass: input data flows through the network layer by layer, each layer applying a weighted sum followed by an activation function, until the network produces an output. That output is compared to the true label using a loss function — mean squared error for regression, cross-entropy for classification — producing a single scalar number that measures how wrong the prediction was. Everything backpropagation does afterward is aimed at figuring out how to nudge every weight to make that scalar smaller next time.
Cricket analogy: The ball travels through the bowler's full action — run-up, load-up, release, follow-through — and the outcome (wicket, boundary, dot ball) is scored against the expected result, producing one number: how costly that delivery was.
The Backward Pass and the Chain Rule
The backward pass computes the gradient of the loss with respect to every weight, starting from the output layer and working toward the input layer. Because the network is a composition of functions (linear transform, then activation, layer after layer), the chain rule lets us express the gradient of the loss with respect to an early-layer weight as a product of local derivatives along the path from that weight to the output. Backpropagation computes these products efficiently by caching the derivative at each layer during the backward sweep, rather than recomputing shared sub-expressions repeatedly — this is what makes it feasible at scale. Each weight then receives a gradient value representing its share of responsibility for the loss.
Cricket analogy: A coach reviewing a lost match works backward from the final score to the last over, then the middle overs, then the powerplay, reusing each phase's noted mistake rather than re-analyzing the footage from scratch each time.
Weight Updates and Learning Rate
Once gradients are known, an optimizer (plain gradient descent, or more commonly a variant like SGD with momentum, RMSProp, or Adam) updates each weight by subtracting a fraction of its gradient, scaled by the learning rate: w_new = w_old - learning_rate * gradient. Too large a learning rate can cause the loss to oscillate or diverge; too small a learning rate makes training crawl and can get stuck in poor local regions. This forward-pass, backward-pass, weight-update cycle repeats for many iterations (epochs) over the training data until the loss converges to an acceptably low value or stops improving on a validation set.
Cricket analogy: A captain adjusts the field placement by a small margin each over based on how the batter is scoring; move the fielders too aggressively and the batter exploits the gaps, too cautiously and singles keep trickling through.
import numpy as np
# Minimal backprop for a single hidden-layer network on toy data
np.random.seed(0)
X = np.random.randn(200, 3)
y = (X[:, 0] + 2 * X[:, 1] - X[:, 2] > 0).astype(float).reshape(-1, 1)
W1 = np.random.randn(3, 4) * 0.5
b1 = np.zeros((1, 4))
W2 = np.random.randn(4, 1) * 0.5
b2 = np.zeros((1, 1))
lr = 0.1
def sigmoid(z):
return 1 / (1 + np.exp(-z))
for epoch in range(2000):
# forward pass
z1 = X @ W1 + b1
a1 = np.maximum(0, z1) # ReLU hidden layer
z2 = a1 @ W2 + b2
a2 = sigmoid(z2) # sigmoid output
loss = -np.mean(y * np.log(a2 + 1e-9) + (1 - y) * np.log(1 - a2 + 1e-9))
# backward pass (chain rule)
dz2 = (a2 - y) / len(X)
dW2 = a1.T @ dz2
db2 = dz2.sum(axis=0, keepdims=True)
da1 = dz2 @ W2.T
dz1 = da1 * (z1 > 0) # ReLU derivative
dW1 = X.T @ dz1
db1 = dz1.sum(axis=0, keepdims=True)
# gradient descent update
W1 -= lr * dW1; b1 -= lr * db1
W2 -= lr * dW2; b2 -= lr * db2
print(f'final loss: {loss:.4f}')A useful mental model: backpropagation is 'blame assignment.' The final loss is a single number of disappointment, and backprop traces that disappointment backward through every weight in the network, computing exactly how much of the blame each weight deserves — and therefore how much it should change.
A frequent misconception is that backpropagation IS the optimization algorithm. It isn't — backpropagation only computes gradients efficiently. Gradient descent (or Adam, RMSProp, etc.) is the separate step that actually uses those gradients to update the weights.
- Backpropagation efficiently computes the gradient of the loss with respect to every weight using the chain rule.
- Training alternates a forward pass (compute predictions and loss) with a backward pass (compute gradients).
- Backprop caches intermediate derivatives during the backward sweep instead of recomputing them, which is what makes it fast at scale.
- Gradients are consumed by an optimizer (SGD, Adam, RMSProp) that updates weights using the learning rate.
- Too high a learning rate causes divergence; too low a learning rate causes painfully slow convergence.
- Backpropagation computes gradients; it does not itself decide how large a weight update to make — that's the optimizer's job.
Practice what you learned
1. What mathematical rule does backpropagation rely on to compute gradients across multiple layers?
2. In the standard training loop, what happens immediately after the forward pass computes the loss?
3. What is the main efficiency advantage of backpropagation over naively perturbing each weight individually to estimate its gradient?
4. What is likely to happen if the learning rate is set far too high during training?
5. Which statement correctly distinguishes backpropagation from gradient descent?
Was this page helpful?
You May Also Like
Activation Functions
Activation functions inject non-linearity into neural networks, letting them approximate complex functions instead of collapsing into a single linear transformation.
What Is a Neural Network?
An introduction to neural networks as layered compositions of weighted sums and nonlinear activations, and how they learn through forward passes and training.
Gradient Descent Explained
Understand how gradient descent iteratively adjusts model parameters to minimize a loss function, and how learning rate and variants like SGD affect convergence.
Hyperparameter Tuning
Explore systematic strategies — grid search, random search, and cross-validation-based tuning — for choosing model settings that are not learned directly from data.