How Does Gradient Descent Work?
Understand gradient descent: the update rule, learning rate, batch vs stochastic vs mini-batch variants, and a from-scratch Python linear regression example.
Expected Interview Answer
Gradient descent is an iterative optimization algorithm that minimises a loss function by repeatedly nudging the model's parameters in the direction of the negative gradient — the steepest downhill direction — scaled by a learning rate, until it reaches a minimum.
At each step it computes the gradient of the loss with respect to every parameter, then updates the parameters with theta = theta - learning_rate * gradient. The learning rate controls step size: too large and it overshoots or diverges, too small and it crawls. Variants differ in how much data they use per step — batch gradient descent uses the whole dataset, stochastic gradient descent (SGD) uses one sample, and mini-batch (the common default) uses small groups — trading gradient accuracy against speed and noise. Because it only follows local slope, on non-convex loss surfaces it can settle in local minima or saddle points, which is why momentum and adaptive optimizers like Adam are used.
- Scales to millions of parameters and huge datasets
- Works for any differentiable loss function
- Mini-batch variants exploit fast vectorised hardware
- Foundation of training for nearly all neural networks
- Tunable trade-off between speed, noise and accuracy
AI Mentor Explanation
Gradient descent is a batter adjusting technique between deliveries to minimise dismissals. Each ball is feedback: the coach points out the biggest flaw (the gradient), and the batter makes a small correction (the learning rate) in that direction. Correct too aggressively and you overcompensate into a new fault; too timidly and you never improve. Over many balls the technique settles into a groove — the loss minimum.
Step-by-Step Explanation
Step 1
Initialise parameters
Start weights at small random values (or zeros for simple linear models) as the starting point on the loss surface.
Step 2
Compute the loss
Run a forward pass and measure the loss function comparing predictions to the true targets.
Step 3
Compute the gradient
Use backpropagation/calculus to get the partial derivative of the loss with respect to each parameter.
Step 4
Update the parameters
Apply theta = theta - learning_rate * gradient, moving each parameter downhill against its gradient.
Step 5
Repeat until convergence
Loop over epochs until the loss stops improving meaningfully or a step/iteration budget is reached.
What Interviewer Expects
- The update rule theta = theta - lr * gradient
- Why we move in the negative gradient direction
- The role and risks of the learning rate
- Difference between batch, stochastic and mini-batch GD
- Awareness of local minima, saddle points and momentum/Adam
Common Mistakes
- Moving in the positive gradient direction (that maximises loss)
- Assuming a bigger learning rate always trains faster
- Confusing an epoch with a single parameter update
- Ignoring feature scaling, which distorts the loss surface
- Believing gradient descent always finds the global minimum
Best Answer (HR Friendly)
“Gradient descent is how a model learns by trial and correction: it checks how wrong it is, figures out which way to tweak its settings to be less wrong, and takes a small step in that direction. Repeating this many times gradually lands the settings on values that give the best predictions.”
Code Example
import numpy as np
# y = 2x + 1 with a little noise
np.random.seed(0)
X = np.random.rand(100, 1)
y = 2 * X + 1 + 0.05 * np.random.randn(100, 1)
w, b = 0.0, 0.0 # parameters
lr = 0.1 # learning rate
n = len(X)
for epoch in range(1000):
y_pred = w * X + b
error = y_pred - y
# gradients of mean squared error
dw = (2 / n) * np.sum(error * X)
db = (2 / n) * np.sum(error)
# update rule: step against the gradient
w -= lr * dw
b -= lr * db
print(f"Learned w={w:.3f}, b={b:.3f}") # ~2.0 and ~1.0Follow-up Questions
- How do batch, stochastic and mini-batch gradient descent differ?
- What problems does momentum solve in gradient descent?
- How do adaptive optimizers like Adam improve on plain SGD?
- What happens if the learning rate is too high or too low?
- Why does feature scaling help gradient descent converge faster?
MCQ Practice
1. The gradient descent update rule is:
Parameters move against the gradient (downhill), scaled by the learning rate: theta = theta - lr * gradient.
2. Which variant updates parameters using a single training example per step?
Stochastic gradient descent (SGD) updates on one sample at a time, giving noisy but fast updates.
3. A learning rate that is too large typically causes:
Large steps can jump past the minimum and cause the loss to oscillate or blow up.
Flash Cards
Gradient descent update rule — theta = theta - learning_rate * gradient — step against the gradient.
Why the negative gradient? — The gradient points uphill (increasing loss); the negative direction decreases loss fastest.
Batch vs SGD vs mini-batch — Whole dataset vs one sample vs small groups per update — accuracy vs speed/noise trade-off.
Learning rate too high vs too low — Too high overshoots/diverges; too low converges very slowly.