Adam Optimizer
Adam (Adaptive Moment Estimation) is a gradient-based optimization algorithm that maintains per-parameter adaptive learning rates using running estimates of the first and second moments of the gradients.
Definition
Adam (Adaptive Moment Estimation) is a gradient-based optimization algorithm that maintains per-parameter adaptive learning rates using running estimates of the first and second moments of the gradients.
Overview
Adam was introduced by Diederik Kingma and Jimmy Ba in a 2014 paper and has become the default optimizer for training most deep neural networks, including the majority of modern large language models. It combines ideas from two earlier optimizers: momentum, which smooths the gradient direction using an exponentially decaying average of past gradients (the first moment), and RMSprop, which adapts the effective learning rate for each parameter using an exponentially decaying average of past squared gradients (the second moment). For each parameter, Adam computes bias-corrected estimates of these first and second moments and uses their ratio to scale the update, meaning parameters with consistently large gradients get smaller effective steps while parameters with small or infrequent gradients get relatively larger steps. This per-parameter adaptivity, combined with momentum's smoothing effect, makes Adam relatively robust to the choice of initial learning rate and effective on the sparse, noisy gradients typical of large-scale deep learning, which is a major reason for its widespread adoption over plain stochastic gradient descent. A well-known refinement is AdamW, introduced by Ilya Loshchilov and Frank Hutter, which decouples weight decay (L2 regularization) from the gradient-based update, fixing a subtle interaction in the original Adam formulation where adaptive scaling distorted the effect of weight decay. AdamW is now the default optimizer for training transformer-based language models such as GPT and BERT variants. Adam's main hyperparameters are the learning rate, the two moment decay rates (beta1 and beta2, commonly 0.9 and 0.999), and a small epsilon term for numerical stability.
Key Concepts
- Combines momentum (first moment) and RMSprop-style adaptive scaling (second moment)
- Maintains per-parameter adaptive learning rates rather than one global rate
- Uses bias correction to account for the moments being initialized at zero
- Robust to the choice of initial learning rate compared to plain SGD
- Effective on sparse gradients and noisy large-scale training objectives
- AdamW variant decouples weight decay from the adaptive gradient update
- Default optimizer for training most transformer-based language models
- Key hyperparameters: learning rate, beta1, beta2, and epsilon