Adam Optimizer
Adam (Adaptive Moment Estimation) is a gradient-based optimization algorithm that maintains per-parameter adaptive learning rates using running estimates of the first and second moments of the gradients.
6 resources across 1 library
Glossary Terms(6)
Layer Normalization
Layer normalization is a neural network normalization technique that normalizes activations across the features of a single training example, rather than acros…
L2 Regularization
L2 regularization is a technique that adds a penalty proportional to the sum of the squared values of a model's weights to its loss function, discouraging larg…
Learning Rate Scheduling
Learning rate scheduling is the practice of systematically adjusting a model's learning rate during training, rather than keeping it fixed, to speed up converg…
Adam Optimizer
Adam (Adaptive Moment Estimation) is a gradient-based optimization algorithm that maintains per-parameter adaptive learning rates using running estimates of th…
Stochastic Gradient Descent
Stochastic gradient descent (SGD) is an iterative optimization algorithm that updates model parameters using the gradient of the loss computed on a small, rand…
Cross-Entropy Loss
Cross-entropy loss is a loss function that measures the difference between a predicted probability distribution and the true (target) distribution, commonly us…