What is Gradient Descent?
Learn what gradient descent is, how it minimizes loss functions, its batch/SGD/mini-batch variants, and how learning rate affects convergence.
Expected Interview Answer
Gradient descent is an iterative optimization algorithm that minimizes a loss function by repeatedly stepping in the direction opposite to the gradient (the steepest increase) of that loss with respect to the model's parameters, gradually moving toward a minimum.
At each step, gradient descent computes the gradient of the loss function with respect to every parameter, then updates each parameter by subtracting the gradient scaled by a learning rate, so parameters move slightly downhill toward lower loss. The learning rate controls the step size: too large and the algorithm overshoots or diverges, too small and convergence is painfully slow. Variants include batch gradient descent (uses the full dataset each step), stochastic gradient descent or SGD (uses one example at a time, noisy but fast), and mini-batch gradient descent (a practical middle ground used almost universally in deep learning). More advanced optimizers like Adam or RMSprop adapt the effective learning rate per parameter using running estimates of gradient magnitude, generally converging faster and more reliably than plain SGD.
- General-purpose optimizer that works for almost any differentiable loss function
- Scales to millions of parameters, as used in deep neural networks
- Mini-batch and stochastic variants make it computationally efficient
- Well-understood convergence behavior with tunable learning rate
- Foundation for advanced optimizers like Adam and RMSprop
AI Mentor Explanation
Gradient descent is like a batsman adjusting their stance delivery by delivery, feeling which small tweak reduces mistimed shots the most and nudging technique slightly in that direction each time. The learning rate is how big each adjustment is: too large a change and the stance overcorrects wildly, too small and it takes forever to find the ideal balance.
Step-by-Step Explanation
Step 1
Initialize parameters
Start with random or zero-initialized weights for the model.
Step 2
Compute the loss
Run a forward pass to compute predictions and measure how far they are from targets using a loss function.
Step 3
Compute the gradient
Use calculus (backpropagation for neural nets) to compute the partial derivative of the loss with respect to each parameter.
Step 4
Update parameters
Subtract the gradient scaled by the learning rate from each parameter, moving it toward lower loss.
Step 5
Repeat until convergence
Iterate the process over batches or epochs until the loss stops meaningfully decreasing.
Step 6
Tune the learning rate and variant
Choose batch, mini-batch, or stochastic gradient descent, and consider adaptive optimizers like Adam for faster, more stable convergence.
What Interviewer Expects
- Explains gradient descent as iteratively stepping opposite the gradient to minimize loss
- Correctly describes the role and risk of the learning rate
- Distinguishes batch, stochastic, and mini-batch variants
- Can mention at least one advanced optimizer like Adam
- Understands that gradient descent can converge to a local minimum, not always global
Common Mistakes
- Confusing the gradient direction — forgetting you subtract it, not add it
- Ignoring the effect of learning rate on convergence and divergence
- Claiming gradient descent always finds the global minimum
- Not knowing the difference between batch and stochastic gradient descent
- Forgetting that gradient descent requires a differentiable loss function
Best Answer (HR Friendly)
“Gradient descent is a method a computer uses to gradually improve a model by making small adjustments that reduce its errors, repeating this many times until the errors are as small as possible. It's like slowly tuning knobs step by step until you get the best possible result.”
Code Example
import numpy as np
# Simple linear regression: minimize (y - w*x - b)^2
X = np.array([1.0, 2.0, 3.0, 4.0])
y = np.array([3.0, 5.0, 7.0, 9.0])
w, b = 0.0, 0.0
learning_rate = 0.01
for epoch in range(1000):
y_pred = w * X + b
error = y_pred - y
grad_w = np.mean(2 * error * X)
grad_b = np.mean(2 * error)
w -= learning_rate * grad_w
b -= learning_rate * grad_b
print(f"Learned: w={w:.2f}, b={b:.2f}") # approx w=2.0, b=1.0import torch
model = torch.nn.Linear(1, 1)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
loss_fn = torch.nn.MSELoss()
for epoch in range(100):
optimizer.zero_grad()
predictions = model(X_batch)
loss = loss_fn(predictions, y_batch)
loss.backward() # computes gradients
optimizer.step() # updates parametersFollow-up Questions
- What is the difference between batch, stochastic, and mini-batch gradient descent?
- How does the learning rate affect convergence, and how do you choose it?
- What is the Adam optimizer and how does it improve on plain SGD?
- What happens if the loss function is non-convex?
- What is a learning rate schedule and why is it used?
MCQ Practice
1. In gradient descent, parameters are updated by moving:
Gradient descent subtracts the gradient (scaled by the learning rate) to move opposite the direction of steepest increase, minimizing the loss.
2. What happens if the learning rate is set too high?
A learning rate that is too large causes updates to overshoot the minimum, potentially causing the loss to oscillate or diverge.
3. Which gradient descent variant updates parameters using one training example at a time?
Stochastic gradient descent (SGD) computes the gradient and updates parameters using a single example at a time, making it noisy but fast.
Flash Cards
What does gradient descent minimize? — A loss function, by iteratively moving parameters opposite to the gradient direction.
What does the learning rate control? — The size of each parameter update step; too large risks divergence, too small is slow to converge.
Name the three main gradient descent variants. — Batch, stochastic (SGD), and mini-batch gradient descent.
Name an adaptive optimizer built on gradient descent. — Adam — it adapts the effective learning rate per parameter using running gradient statistics.