Stochastic Gradient Descent
Stochastic gradient descent (SGD) is an iterative optimization algorithm that updates model parameters using the gradient of the loss computed on a small, randomly sampled batch of training data rather than the entire dataset at once.
8 resources across 1 library
Glossary Terms(8)
Batch Normalization
Batch normalization is a neural network training technique that normalizes the inputs to each layer using the mean and variance computed over a mini-batch, the…
L1 Regularization
L1 regularization is a technique that adds a penalty proportional to the sum of the absolute values of a model's weights to its loss function, encouraging spar…
L2 Regularization
L2 regularization is a technique that adds a penalty proportional to the sum of the squared values of a model's weights to its loss function, discouraging larg…
Learning Rate Scheduling
Learning rate scheduling is the practice of systematically adjusting a model's learning rate during training, rather than keeping it fixed, to speed up converg…
Adam Optimizer
Adam (Adaptive Moment Estimation) is a gradient-based optimization algorithm that maintains per-parameter adaptive learning rates using running estimates of th…
Stochastic Gradient Descent
Stochastic gradient descent (SGD) is an iterative optimization algorithm that updates model parameters using the gradient of the loss computed on a small, rand…
Cross-Entropy Loss
Cross-entropy loss is a loss function that measures the difference between a predicted probability distribution and the true (target) distribution, commonly us…
Mean Squared Error
Mean squared error (MSE) is a loss function that measures the average of the squared differences between predicted and actual values, commonly used to train an…