How deeply does Deep Learning with R, Third Edition cover model parallelism?

Explains the device split and shows mesh, variable-layout, and strategy configuration; stops short of deriving distributed computation internals.

Model parallelism

As explained in Deep Learning with R, Third Edition

Model parallelism is a distributed-training strategy that partitions a model across devices so different devices execute different model components.

Model parallelism splits one model across devices. Different devices run different parts of the model while processing a batch together. Its practical purpose is to train a model too large to fit on one device—not to speed up an ordinary model that already fits. section 18.2.1

Contrast this with data parallelism: each device holds a complete model replica and processes a different portion of the batch. The two approaches can be combined. For example, eight GPUs can hold two replicas of a model that is itself split across four GPUs. section 18.2.1

Split the work, then replicate it

Picture a 2 × 4 grid of GPUs. Along the four-device “model” axis, devices share one model’s computation. Along the two-device “data” axis, two copies of that split model process different sub-batches. This arrangement gives a model more room without giving up data parallelism. section 18.2.1 section 18.2.2

device_mesh <- keras$distribution$DeviceMesh(
  shape = shape(2, 4),  #1
  axis_names = c("data", "model")  #2
)

In Keras, a DeviceMesh names those two axes and gives them sizes: two for data, four for model. The mesh organizes devices; it does not yet specify which model variables to split. section 18.2.2

layout_map <- keras$distribution$LayoutMap(device_mesh)
layout_map["sequential/dense/kernel"]   <- tuple(NULL, "model")
layout_map["sequential/dense/bias"]     <- tuple("model")
layout_map["sequential/dense_1/kernel"] <- tuple(NULL, "model")
layout_map["sequential/dense_1/bias"]   <- tuple("model")

A LayoutMap assigns variable layouts on that mesh. Here, dense-layer kernels and biases are mapped to the model axis. Match the paths to the actual model variables; they can be printed from model$variables. section 18.2.2

model_parallel <- keras$distribution$ModelParallel(
    layout_map=layout_map,
    batch_dim_name="data"  #1
)
keras$distribution$set_distribution(model_parallel)

ModelParallel combines the layout map with the data-axis name. Set it as the active Keras distribution strategy before using the model under that strategy. section 18.2.2

Choose model parallelism when model size demands it, especially if the architecture has naturally parallel branches. If the entire model fits on each device, data parallelism is the simpler scaling choice. section 18.2.1 section 18.4

How Deep Learning with R, Third Edition covers it

Cover of Deep Learning with R, Third Edition

liveBook manning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Designing Deep Learning Systems: A guide for software engineersModel parallelism is a distributed training strategy that partitions a model's computation or parameters across multiple devices.
liveBook manning.com

Deep Learning with Python, Third EditionA distributed-training method where different parts of one model are placed on different devices to fit or execute a larger model.
liveBook manning.com

Sutskever's ListModel parallelism is a technique that partitions a neural network across multiple computational devices instead of replicating the whole model on each device.
liveBook manning.com

Deep Learning with PyTorch, Second EditionA strategy that places different parts of a model on different devices.

Compare how 5 books explain model parallelism →

Related concepts

distributed trainingkind of · Distributed training is the coordinated training of a machine-learning model across multiple computing devices, processes, or machines by distributing computation or data among them.
pipeline parallelismincludes · Pipeline parallelism is a model-parallel technique that divides sequential computation into stages on different devices and concurrently processes multiple inputs through them.
data parallelcontrasts with · Data parallelism is a parallel-computing technique that applies the same computation concurrently to separate portions of data.
AlexNetused by · AlexNet is a convolutional neural network architecture designed for image classification and recognized for its landmark performance in large-scale image recognition.
fully sharded data parallelused by · Fully Sharded Data Parallel is a distributed training technique that partitions model parameters, gradients, and optimizer states across data-parallel devices to reduce per-device memory use.
GPUuses · A graphics processing unit (GPU) is a processor designed for highly parallel computation, especially graphics and data-parallel workloads.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.