liveBook · Manning

Fully sharded data parallel

Also called FSDP, fully sharded data parallelism

Fully sharded data parallel is a distributed training strategy that partitions model parameters, gradients, and optimizer states across data-parallel workers to reduce per-worker memory use.

How 2 Manning books explain it

CUDA for LLMs

Fully sharded data parallel is a distributed training technique that shards model parameters, gradients, and optimizer states across devices to reduce per-device memory use.

How it is used
Uses FSDP to train models whose replicated training state exceeds single-GPU memory.
How deep it goes
Explains state sharding, temporary weight reconstruction, and gradient distribution, with per-GPU memory accounting.
Examples
Calculates the training-state memory for a 7B-parameter model on eight GPUs and illustrates ReduceScatter with four GPUs.
What you can do after
Evaluate when sharding training state can make a model fit across GPUs and identify the collective communication involved.

Read how CUDA for LLMs explains fully sharded data parallel →

Deep Learning with PyTorch, Second Edition

A distributed-training method that shards model state across devices to reduce memory use.

How it is used
Scales models beyond one GPU by distributing model state, data, computation, and communication across devices.
How deep it goes
Explains parameter sharding, all_gather, reduce_scatter, communication overlap, device meshes, and module-level configuration.
Examples
Uses a two-device mesh, linear-layer sharding, root-model sharding, and pseudocode illustrating FSDP implementation.
What you can do after
Operate/configure FSDP with torch.distributed.fsdp, fully_shard, and DeviceMesh for selected modules and complete models.

Read how Deep Learning with PyTorch, Second Edition explains fully sharded data parallel →

Also covered in

What to know first

Related concepts

Questions about fully sharded data parallel

What is fully sharded data parallel?

Fully sharded data parallel is a distributed training strategy that partitions model parameters, gradients, and optimizer states across data-parallel workers to reduce per-worker memory use.

Which Manning books explain fully sharded data parallel?

CUDA for LLMs and Deep Learning with PyTorch, Second Edition.

What should I know before learning fully sharded data parallel?

Manning books that teach fully sharded data parallel usually introduce model parallelism first.

What is fully sharded data parallel a kind of?

Fully sharded data parallel is a kind of data parallelism. Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.

What is fully sharded data parallel also called?

Fully sharded data parallel is also called FSDP or fully sharded data parallelism.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.