Fully sharded data parallel
Fully sharded data parallel is a distributed training strategy that partitions model parameters, gradients, and optimizer states across data-parallel workers to reduce per-worker memory use.
CUDA for LLMs
Fully sharded data parallel is a distributed training technique that shards model parameters, gradients, and optimizer states across devices to reduce per-device memory use.
- How it is used
- Uses FSDP to train models whose replicated training state exceeds single-GPU memory.
- How deep it goes
- Explains state sharding, temporary weight reconstruction, and gradient distribution, with per-GPU memory accounting.
- Examples
- Calculates the training-state memory for a 7B-parameter model on eight GPUs and illustrates ReduceScatter with four GPUs.
- What you can do after
- Evaluate when sharding training state can make a model fit across GPUs and identify the collective communication involved.
Read how CUDA for LLMs explains fully sharded data parallel →
Deep Learning with PyTorch, Second Edition
A distributed-training method that shards model state across devices to reduce memory use.
- How it is used
- Scales models beyond one GPU by distributing model state, data, computation, and communication across devices.
- How deep it goes
- Explains parameter sharding, all_gather, reduce_scatter, communication overlap, device meshes, and module-level configuration.
- Examples
- Uses a two-device mesh, linear-layer sharding, root-model sharding, and pseudocode illustrating FSDP implementation.
- What you can do after
- Operate/configure FSDP with torch.distributed.fsdp, fully_shard, and DeviceMesh for selected modules and complete models.
Read how Deep Learning with PyTorch, Second Edition explains fully sharded data parallel →
Also covered in
What to know first
- model parallelismModel parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
Related concepts
- data parallelismkind of · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
- all-gatheruses · All-gather is a collective communication operation that gathers data from every participant and makes the combined result available to all participants.
- reduce_scatteruses · Reduce-scatter is a collective communication operation that reduces corresponding values across participants and distributes distinct portions of the result among them.
- model parallelismuses · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
- QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that combines a quantized pretrained model with trainable low-rank adapters.
- distributed trainingimplements · Distributed training is the coordinated execution of machine-learning model training across multiple computing devices, processes, or machines.
Questions about fully sharded data parallel
What is fully sharded data parallel?
Fully sharded data parallel is a distributed training strategy that partitions model parameters, gradients, and optimizer states across data-parallel workers to reduce per-worker memory use.
Which Manning books explain fully sharded data parallel?
CUDA for LLMs and Deep Learning with PyTorch, Second Edition.
What should I know before learning fully sharded data parallel?
Manning books that teach fully sharded data parallel usually introduce model parallelism first.
What is fully sharded data parallel a kind of?
Fully sharded data parallel is a kind of data parallelism. Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
What is fully sharded data parallel also called?
Fully sharded data parallel is also called FSDP or fully sharded data parallelism.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

