FSDP

Definition

Fully Sharded Data Parallel (FSDP) is a distributed training strategy built into PyTorch that shards a model's parameters, gradients, and optimizer states across multiple GPUs instead of replicating full copies on each device. It allows models too large to fit in a single GPU's memory to be trained by having each device hold only a fraction of the total state and gathering the pieces it needs on demand.

Overview

Standard data-parallel training replicates the entire model on every GPU and only splits the input data, which means memory requirements per device do not shrink no matter how many GPUs are added. FSDP was introduced into PyTorch to break that constraint by sharding the model's parameters, gradients, and optimizer states across the participating devices, so adding more GPUs directly reduces the memory each one must hold, not just the batch size each one processes. Mechanically, FSDP wraps a model's layers so that each device stores only its assigned shard of parameters at rest. During the forward and backward passes, it temporarily gathers the full parameters for the layer currently being computed via an all-gather communication step, runs the computation, and then discards the full copy again, keeping only its shard. Gradients are similarly reduced and re-sharded rather than kept in full on every device. This trades some additional communication overhead for a large reduction in peak memory per GPU. FSDP is PyTorch's native answer to the same problem DeepSpeed's ZeRO addresses, and the two share conceptual roots; in fact, FSDP was influenced directly by ZeRO's design. The practical difference is integration: FSDP ships as part of PyTorch itself and is designed to work with minimal extra dependencies and tight compatibility with other PyTorch-native tooling, while DeepSpeed offers a more extensive and separately maintained feature set, including deeper offloading options. Megatron-LM's tensor and pipeline parallelism address a related but distinct axis of scaling and are sometimes combined with FSDP. In practice, teams use FSDP to train and fine-tune large language models and other large networks when they want to stay within the standard PyTorch ecosystem rather than adopting a separate training framework. It has become a common choice for large-scale training recipes published by research groups that build directly on PyTorch, and it is increasingly the default recommendation for multi-GPU or multi-node training of large models in PyTorch-based codebases. The trade-offs center on communication overhead: the constant gathering and re-sharding of parameters requires fast interconnects between GPUs, and poor network bandwidth can turn FSDP's memory savings into a throughput bottleneck. Configuring sharding strategy, wrapping policy, and mixed precision correctly requires some expertise, and smaller models that already fit in GPU memory generally see no benefit and may run slower than plain data parallelism due to the added communication. Debugging distributed training issues can also be harder than with a single-device setup, since failures may only surface under specific sharding configurations or at particular cluster scales, making careful incremental testing important before committing to a large, expensive training run.

Key Features

• Shards model parameters, gradients, and optimizer states across GPUs
• Gathers full parameters temporarily only for the layer being computed
• Built natively into PyTorch rather than as a separate framework
• Reduces per-GPU memory as more devices are added to training
• Supports mixed-precision training alongside sharding
• Conceptually related to DeepSpeed's ZeRO optimizer sharding
• Configurable wrapping policies control which layers are sharded together
• Commonly used for training and fine-tuning large models in PyTorch-native codebases

Use Cases

Alternatives

Frequently Asked Questions

Related in AI Models