A grid-like arrangement of devices used to assign different parallelism roles along named axes.
A device mesh is a grid of devices whose named axes assign different parallelism roles. One axis commonly handles data parallelism, while another handles model parallelism. section 18.2.2
An eight-GPU mesh
Arrange eight GPUs as a 2 × 4 mesh with axes named data and model. The four model-axis devices split computation across the model, while the two data-axis positions run separate sub-batches in parallel. section 18.2.2
Practical rules:
Use one-dimensional or two-dimensional meshes in common practice; higher-dimensional shapes are possible. section 18.2.2
Name axes according to their intended roles, then use a LayoutMap to assign variables across those axes. section 18.2.2
How Deep Learning with Python, Third Edition covers it
How deep it goes
Defines grid shapes, axis roles, and a 2 × 4 example before handing variable placement to LayoutMap.
Examples
Uses an eight-GPU 2 × 4 grid, named data and model axes, and Keras DeviceMesh construction.
What you can do after
Configure a Keras DeviceMesh that assigns separate axes to data and model parallelism.
How it connects
Connects device meshes with data parallelism, model parallelism, LayoutMap, and variable sharding.
Deep Learning with JAXAn arrangement of multiple devices in a grid used to assign different tensor shards to different devices for parallel computation.
dataset splitused by · A dataset split is the division of a dataset into distinct subsets assigned different purposes, such as training, validation, or testing.
model parallelismused by · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
data paralleluses · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
distributed traininguses · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.
