Distributed Training: Horovod and PyTorch DDP

Modern models and datasets routinely outgrow a single GPU: training can take days, or the model simply will not fit in one device's memory, and distributed training exists to spread that work across many GPUs and machines so it finishes in hours and scales beyond any single accelerator. The dominant approach, data parallelism, replicates the model on each GPU, feeds each a different slice of the batch, and synchronises gradients so all replicas stay identical. PyTorch's DistributedDataParallel and Uber's Horovod are the two standard implementations, both built on efficient collective communication. The problem they solve is concrete: naive single-GPU training caps your throughput at one device, and crude approaches like parameter servers bottleneck on a central node. DDP and Horovod instead use all-reduce, a decentralised pattern where every GPU contributes and receives the averaged gradient, scaling near-linearly with hardware. Understanding distributed training is essential because at any serious scale, the question is not whether to distribute but how to do so without communication overhead erasing the speedup.