Custom training loops with tf.GradientTape — Deep Learning & Neural Networks

model.fit() is excellent for standard training scenarios, but it abstracts away control that advanced use cases require. Custom training loops using tf.GradientTape give you complete control over every aspect of training: computing gradients for only a subset of model parameters, implementing multiple interleaved loss functions (as in GANs), applying gradient transformations before the update, training models that require multiple forward passes per batch (meta-learning), or integrating custom regularisation terms that are not standard Keras losses. tf.GradientTape works by recording all TensorFlow operations on watched variables within its context, then computing exact analytical gradients via reverse-mode automatic differentiation. Every call to model.fit() ultimately uses the same GradientTape mechanism internally — understanding it directly removes all magic from the training process and makes debugging trivial. Production training pipelines at Google, DeepMind, and OpenAI are all implemented using GradientTape or its PyTorch equivalent rather than high-level fit() loops.

Analogy🏏Cricket

🏏 Think of it like cricket: The Duckworth-Lewis-Stern (DLS) method, the ICC bowling economy metric, and the net run rate each measure team performance but optimise for completely different things — DLS cares about wickets-in-hand, economy cares about runs-per-over, NRR cares about tournament-wide margins. Using the wrong metric to evaluate a bowler is like using MSE for classification: technically computable but optimising the wrong thing entirely. Just as you would never rank a spinner by batting average, you should never use MSE when your model predicts probabilities. Just as focal DLS adjustments down-weight easy chases and amplify close finishes, focal loss down-weights easy examples and amplifies hard minority-class examples. The choice of scoring metric defines what excellence means — and so does the choice of loss function.