Gradient Descent
Gradient descent is an optimization algorithm that iteratively adjusts a model's parameters in the direction that most reduces its error, or loss, in order to train it.
Definition
Gradient descent is an optimization algorithm that iteratively adjusts a model's parameters in the direction that most reduces its error, or loss, in order to train it.
Overview
At each training step, gradient descent computes the gradient — the direction and rate of steepest increase — of a loss function with respect to the model's parameters, then updates the parameters a small step in the opposite direction to reduce the loss. Repeated over many iterations across a training dataset, this process gradually moves the model toward parameter values that minimize prediction error. In deep learning, the gradients needed for this process are computed efficiently across all layers of a neural network using backpropagation, which applies the chain rule to propagate the error signal from the output layer back to the earliest layers. The size of each parameter update step is controlled by the learning rate, a critical hyperparameter: too large and training can become unstable or diverge, too small and training can be prohibitively slow. In practice, most modern training uses variants like stochastic gradient descent (SGD), which updates parameters using small random batches of data rather than the full dataset, and adaptive optimizers such as Adam, which adjust the effective learning rate per parameter. Gradient descent is the foundational optimization method behind training virtually all modern neural networks, from small classifiers to large-scale foundation models.
Key Concepts
- Iteratively updates model parameters to minimize a loss function
- Relies on backpropagation to compute gradients across neural network layers
- Learning rate controls the size of each parameter update step
- Stochastic gradient descent uses random data batches for efficiency
- Adaptive optimizers like Adam adjust learning rates per parameter
- Core optimization method behind training nearly all neural networks
Use Cases
Frequently Asked Questions
From the Blog
What Is Gradient Descent Explained for Beginners
Gradient descent is the algorithm that trains most machine learning models by repeatedly nudging parameters in the direction that reduces error, step by step.
Read More Data ScienceA Practical Feature Engineering Playbook for Tabular Data
Feature engineering for tabular data is best organised by data type and model family, with a validation loop that proves each feature earns its place. Learn how to treat numeric, categorical, temporal and event data differently, avoid leakage, and decide which transformations gradient-boosted trees genuinely need versus which only linear models do.
Read More Data ScienceHow to encode high-cardinality categorical features without blowing up the model
Two axes decide the encoding: how many distinct values the column has, and which model family consumes it. Gradient-boosted trees often need no encoding at all, linear models need one-hot on a reduced vocabulary, and target encoding is safe only when computed out of fold.
Read More Data ScienceHow to fix CUDA out of memory in PyTorch without buying a bigger GPU
GPU memory splits into parameters, gradients, optimiser state and activations, and only the activation term responds to batch size. Measure the breakdown first, then apply remedies in that order: batch size and accumulation, gradient checkpointing, mixed precision, then a leaner optimiser.
Read More