liveBook · Manning

Model parallelism

As explained in Deep Learning with R, Third Edition

Model parallelism is a distributed-training strategy that partitions a model across devices so different devices execute different model components.

Model parallelism splits one model’s variables and computation across devices. The devices work together on a batch, making it possible to train a model that is too large to fit on one device. It is most natural for architectures with parallel branches, and is generally used to accommodate larger models rather than speed up ordinary ones. section 18.2.1 section 18.4

Listing 18.4 A large, densely connected model
model <- keras_model_sequential(input_shape = c(16000)) |>
  layer_dense(64000, activation = "relu") |>
  layer_dense(8000, activation = "sigmoid")

This densely connected model provides an example of a large model whose computation could be split across devices. section 18.2.1

.[half_kernel_0, half_kernel_1] <- op_split(kernel, 2)
.[half_bias_0, half_bias_1] <- op_split(bias, 2)

with(keras$device("gpu:0"), {
  half_output_0 <- op_matmul(inputs, half_kernel_0) + half_bias_0
})

with(keras$device("gpu:1"), {
  half_output_1 <- op_matmul(inputs, half_kernel_1) + half_bias_1
})

Here, the kernel and bias are split in two. Each GPU computes an output using its assigned portions. section 18.2.1

Data parallelism instead gives each device a complete model replica and different data to process. The two strategies can be combined: four devices can share one model, while another four run a second copy on a different sub-batch. section 18.2.1 section 18.2.2

device_mesh <- keras$distribution$DeviceMesh(
  shape = shape(2, 4),  #1
  axis_names = c("data", "model")  #2
)

A Keras device mesh names the axes used for each strategy. This 2 × 4 mesh assigns two positions to data parallelism and four to model parallelism. section 18.2.2

layout_map <- keras$distribution$LayoutMap(device_mesh)
layout_map["sequential/dense/kernel"]   <- tuple(NULL, "model")
layout_map["sequential/dense/bias"]     <- tuple("model")
layout_map["sequential/dense_1/kernel"] <- tuple(NULL, "model")
layout_map["sequential/dense_1/bias"]   <- tuple("model")

A layout map assigns model-variable dimensions to mesh axes. These entries split dense-layer kernels and biases along the model axis. section 18.2.2

model_parallel <- keras$distribution$ModelParallel(
    layout_map=layout_map,
    batch_dim_name="data"  #1
)
keras$distribution$set_distribution(model_parallel)

The ModelParallel configuration uses the layout map for model placement and names the data axis for batch distribution. section 18.2.2

How Deep Learning with R, Third Edition covers it

How deep it goes
Explains device splitting and walks through mesh, layout-map, and distribution configuration.
Examples
Illustrates a large dense model, a two-GPU computation split, and a two-by-four device mesh.
What you can do after
Apply model parallelism via Keras distribution tools using a device mesh and variable layouts.
How it connects
Contrasts data parallelism with model parallelism and combines both through a device mesh.

Also explained in

Compare how 5 books explain model parallelism →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.