How deeply does Deep Learning with Python, Third Edition cover model parallelism?

Works through a dense-layer split and a Keras device-mesh configuration, including variable layouts and sharding inspection.

Model parallelism

As explained in Deep Learning with Python, Third Edition

A distributed-training method where different parts of one model are placed on different devices to fit or execute a larger model.

A model can grow too large to fit on one GPU. Model parallelism addresses that limit by placing different parts of one model on different devices. The devices work together on the same batch. This differs from data parallelism, which puts a complete model replica on each device and gives each replica different data. Use model parallelism primarily to train a larger model, not as the default way to speed up a model that already fits on one device. section 18.2.1 section 18.4

Split a large dense layer

Consider a Keras model with a 16,000-feature input, a Dense layer with 64,000 units, and a second Dense layer with 8,000 units. One way to distribute the first layer is to split its kernel and bias into two halves along the output-unit dimension. Run the matrix multiplication for one half on gpu:0 and the other on gpu:1. Each device computes a different part of that layer’s output from the same input. This illustrates the central choice: divide the model’s work, rather than divide the batch between full copies of the model. section 18.2.1

A practical Keras setup with JAX describes the device arrangement first. A DeviceMesh with shape (2, 4) and axes named data and model uses four devices to split model computation, then makes two copies of that split model for different sub-batches. The two methods can therefore work together: model parallelism handles model size, while data parallelism handles separate data. section 18.2.1 section 18.2.2

device_mesh = keras.distribution.DeviceMesh(
    shape=(2, 4),                                                     #1
    axis_names=["data", "model"],                                     #2
)

The (2, 4) mesh names the two device axes used for data parallelism and model parallelism. section 18.2.2

To specify the split in Keras:

Create a LayoutMap for the device mesh. Map each Dense kernel to (None, "model") and each bias to ("model",), using the variables’ paths as keys. section 18.2.2
Pass the layout map to ModelParallel, set batch_dim_name to "data", and register the distribution strategy. Inspect a variable’s sharding to check the resulting mesh and partition specification. section 18.2.2

Rule of thumb: start with data parallelism when a full model fits on each device; reach for model parallelism when it does not. Architectures with naturally parallel branches are especially well suited to splitting model computation. section 18.2.1 section 18.4

How Deep Learning with Python, Third Edition covers it

Cover of Deep Learning with Python, Third Edition

liveBook manning.com

Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Designing Deep Learning Systems: A guide for software engineersModel parallelism is a distributed training strategy that partitions a model's computation or parameters across multiple devices.
liveBook manning.com

Sutskever's ListModel parallelism is a technique that partitions a neural network across multiple computational devices instead of replicating the whole model on each device.
liveBook manning.com

Deep Learning with R, Third EditionModel parallelism is a distributed-training strategy that partitions a model across devices so different devices execute different model components.
liveBook manning.com

Deep Learning with PyTorch, Second EditionA strategy that places different parts of a model on different devices.

Compare how 5 books explain model parallelism →

Related concepts

distributed trainingkind of · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
pipeline parallelismincludes · Pipeline parallelism is a model-parallel technique that divides sequential computation into stages on different devices and concurrently processes multiple inputs through them.
data parallelcontrasts with · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
fully sharded data parallelused by · Fully Sharded Data Parallel is a distributed training technique that partitions model parameters, gradients, and optimizer states across data-parallel devices to reduce per-device memory use.
AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.
GPUuses · A graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and data-parallel workloads.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.