liveBook · Manning

Mesh-TensorFlow

As explained in Sutskever's List

Mesh-TensorFlow is a framework that expresses distributed tensor computation by mapping tensor axes onto dimensions of a device mesh.

Mesh-TensorFlow generalizes hand-built tensor slicing by letting engineers map tensor axes onto device-mesh dimensions. section 6.3.3

Choosing axes to shard

Sharding the batch gives data parallelism; sharding vocabulary or feed-forward dimensions gives tensor parallelism; sharding a mixture-of-experts model’s expert dimension gives expert parallelism. section 6.3.3

Pipeline parallelism splits computation stages rather than tensor axes, but belongs to the broader mesh family. section 6.3.3

How Sutskever's List covers it

How deep it goes
Explains axis-to-mesh mapping through data, tensor, and expert parallelism examples.
Examples
Uses batch, vocabulary, feed-forward, and mixture-of-experts dimensions as sharding examples.
What you can do after
Apply axis-to-mesh sharding by assigning batch, vocabulary, feed-forward, or expert dimensions across devices.
How it connects
Connects data, tensor, expert, and pipeline parallelism within a broader mesh family.
Cover of Sutskever's List
liveBookmanning.com
Read Sutskever's List on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.