Device mesh
A device mesh is a multidimensional logical arrangement of computational devices organized along named axes for assigning devices to distributed computation and parallelism.
Deep Learning with JAX
An arrangement of multiple devices in a grid used to assign different tensor shards to different devices for parallel computation.
- How it is used
- Deploys device grids to assign tensor shards and coordinate data, tensor, or mixed parallel computation.
- How deep it goes
- Explains topology-aware construction, shape constraints, positional and named mappings, and sharded computation, but defers broader parallelism details.
- Examples
- Demonstrates (8, 1), (1, 8), and (2, 4) meshes for tensor axes, vector dot products, and neural-network training.
- What you can do after
- Operate/configure JAX meshes and sharding specifications for partitioned arrays and mixed-parallel training.
Deep Learning with Python, Third Edition
A grid-like arrangement of devices used to assign different parallelism roles along named axes.
- How it is used
- Organizes devices into named axes for combining data parallelism and model parallelism during distributed training.
- How deep it goes
- Defines grid shapes, axis roles, and a 2 × 4 example before handing variable placement to LayoutMap.
- Examples
- Uses an eight-GPU 2 × 4 grid, named data and model axes, and Keras DeviceMesh construction.
- What you can do after
- Configure a Keras DeviceMesh that assigns separate axes to data and model parallelism.
Read how Deep Learning with Python, Third Edition explains device mesh →
Deep Learning with R, Third Edition
A device mesh is a multidimensional logical arrangement of computational devices organized along named axes.
- How it is used
- Organizes devices for customized data-parallel, model-parallel, or mixed distributed training.
- How deep it goes
- Defines device grids, named axes, and a 2 × 4 construction before assigning variable layouts with LayoutMap.
- Examples
- Uses an eight-GPU grid, a 2 × 4 DeviceMesh, and explicit GPU device names.
- What you can do after
- Operate/configure Keras distribution with a device mesh for model-parallel training and mixed parallel strategies.
Read how Deep Learning with R, Third Edition explains device mesh →
Related concepts
- dataset splitused by · A dataset split is the division of a dataset into distinct subsets assigned different purposes, such as training, validation, or testing.
- model parallelismused by · Model parallelism is a distributed-computing technique that partitions a machine-learning model’s parameters, layers, or computations across multiple devices.
- data paralleluses · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
- distributed traininguses · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.