liveBook · Manning

Rematerialization

As explained in Sutskever's List

Rematerialization is a memory-saving training technique that recomputes intermediate activations instead of storing them throughout the computation.

Rematerialization saves memory during training by recomputing intermediate activations instead of storing them throughout the computation. Paired with model partitioning, it helped GPipe train models beyond the limits of a single device. section 6.3.2 section 6.3.2

Scaling a Transformer

With rematerialization and parameter partitioning, GPipe trained an 83.9-billion-parameter Transformer across 128 TPU partitions. section 6.3.2

How Sutskever's List covers it

How deep it goes
Defines activation recomputation as a memory-saving technique but does not detail its implementation.
Examples
Illustrates the technique with GPipe training an 83.9-billion-parameter Transformer across 128 TPU partitions.
What you can do after
Understand how activation recomputation can help train models too large for a single device.
How it connects
Links rematerialization with model partitioning, pipeline parallelism, and neural-network training.
Cover of Sutskever's List
liveBookmanning.com
Read Sutskever's List on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.