liveBook · Manning

Vertical partitioning

As explained in Deep Learning with R, Third Edition

Vertical partitioning splits each model layer across all available devices.

Vertical partitioning splits every model layer across available devices. Each device performs part of a parallel operation, such as matrix multiplication, then the partial outputs are merged. section 18.2.1

Splitting a dense layer

For a dense layer with 16,000 inputs and 64,000 outputs, split the kernel and bias into halves. GPU 0 computes one half and GPU 1 computes the other; concatenate both outputs to form the complete layer output. section 18.2.1

Unlike horizontal partitioning, which assigns different layers to different devices, vertical partitioning keeps each layer distributed and suits highly parallelizable operations. section 18.2.1

How Deep Learning with R, Third Edition covers it

How deep it goes
Explains kernel and bias splitting, per-device matrix multiplication, output concatenation, and communication trade-offs.
Examples
Uses a densely connected classifier with 16,000 features, 64,000 hidden units, and 8,000 overlapping categories.
What it assumes
Assumes familiarity with model layers, matrix multiplication, GPUs, and the distinction between data and model parallelism.
What you can do after
Apply it via Keras device scopes by splitting layer parameters and combining partial outputs across GPUs.
How it connects
Connects model parallelism with data parallelism, horizontal partitioning, matrix multiplication, convolution, and communication overhead.
Cover of Deep Learning with R, Third Edition
liveBookmanning.com
Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.