Sharding splits model parameters, gradients, or activations into pieces distributed across GPUs or nodes. This reduces memory pressure on individual devices and enables collaborative execution when a model cannot fit on one device or cannot be altered. section 9.2
Table 9.2 Comparison of sharding strategies
|
Strategy
|
Memory savings
|
Communication cost
|
Typical use case
|
|---|---|---|---|
|
Tensor sharding
|
Moderate
|
High
|
Splitting large weight matrices across devices
|
|
Pipeline sharding
|
High
|
Moderate
|
Very deep models with many sequential layers
|
|
Optimizer sharding (ZeRO, FSDP)
|
Very high
|
Moderate to high
|
Training models with hundreds of billions of parameters
|
|
Hybrid sharding
|
Maximal
|
Very high
|
Trillion-parameter scale training with combined strategies
|
Compare tensor, pipeline, optimizer, and hybrid sharding by memory savings, communication cost, and typical use case. section 9.2
Distributing a 175-billion-parameter model
A model requiring 700 GB in FP16 cannot fit on one device; sharding it across 64 GPUs lets each store around 11 GB. During training, sharding can also distribute optimizer states and gradients; during inference, it can distribute layers or experts. section 9.2
