liveBook · Manning

ZeRO

As explained in Sutskever's List

The Zero Redundancy Optimizer is a distributed optimization method that partitions parameters, gradients, and optimizer states across data-parallel processes.

A 50-billion-parameter model requires 100 GB just to store its 16-bit parameters, beyond a single GPU’s capacity. ZeRO, introduced in 2019, reduces this memory burden by partitioning parameters, gradients, and optimizer states across data-parallel devices. Each GPU holds a shard and communicates as needed to recreate the full model when required. section 6.3.3

Scaling across GPUs

A 16-GPU cluster could, in principle, train a model 16 times larger than before. Microsoft demonstrated training a 100-billion-parameter model on 400 GPUs; ZeRO also enabled training a 13-billion-parameter model without model parallelism. section 6.3.3

How Sutskever's List covers it

How deep it goes
Explains partitioning parameters, gradients, and optimizer states, but not specific implementation stages.
Examples
Uses a 16-GPU scaling illustration and demonstrations of 100-billion- and 13-billion-parameter models.
What you can do after
Build larger data-parallel training runs by distributing model-state memory across devices.
How it connects
Connects data parallelism with model parallelism and pipeline parallelism as approaches to scaling model training.
Cover of Sutskever's List
liveBookmanning.com
Read Sutskever's List on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.