Gradient Clipping
Gradient clipping is a training technique that caps the magnitude of gradients during backpropagation, preventing the exploding-gradient problem that can destabilize neural network training, particularly in recurrent networks and deep transformers.
3 resources across 1 library
Glossary Terms(3)
Gradient Clipping
Gradient clipping is a training technique that caps the magnitude of gradients during backpropagation, preventing the exploding-gradient problem that can desta…
Gradient Accumulation
Gradient accumulation is a training technique that simulates a larger batch size than fits in available memory by summing gradients over several smaller mini-b…
Weight Decay
Weight decay is a regularization technique that adds a penalty proportional to the magnitude of a model's weights to the loss function, discouraging overly lar…