In 2012, AlexNet’s eight-layer network had 60 million parameters, but a GTX 580 GPU had only 3 GB of memory—not enough to hold the entire model. Splitting the network across two GPUs made the larger design trainable. The constraint was not just how fast a GPU could compute, but what it could store. section 2.5.4
Model parallelism divides one model among multiple devices rather than placing a complete copy on each. In AlexNet, each GPU handled half the filters in each layer and half the feature maps in a training batch. The halves exchanged information at selected layers: too many connections would make communication costly, while too few would limit how their features could mix. section 2.5.4 section 6.3
Why the split mattered
The two-GPU design allowed twice as many filters per layer as a smaller, single-GPU variant. Its top-1 accuracy improved by 1.7 percentage points and its top-5 accuracy by 1.2 points. It also supported greater depth: removing any convolutional layer and retraining reduced accuracy, despite each such layer containing less than 1% of the parameters. section 2.5.4
The contrast is data parallelism. It gives each GPU a complete model and a different batch, then sums gradients to keep the copies synchronized. That works while a complete model fits on one device; it cannot solve a model-memory limit by itself. Model parallelism instead distributes the weights and computations, introducing decisions about where devices must communicate. section 6.3
One way to make the split is tensor sharding: divide the arrays that hold weights or activations so no device stores them all. AlexNet’s hand-arranged feature-map split was an early version. Another approach, pipeline parallelism, places successive computation stages on different devices. Pipelines can leave devices idle while work enters and leaves them; slicing work within a layer offers a different trade-off. section 6.3 section 6.3.3
From hand-built splits to larger systems
Megatron-LM partitioned matrix multiplications across GPUs and combined their partial results. Unlike a split between layers, this within-layer approach avoided some pipeline idle time, and the two approaches could be combined. Mesh-TensorFlow generalized the choice of which tensor dimensions to distribute across devices. The engineering lesson remained the same as with AlexNet: a useful split must relieve memory pressure without letting communication or idle time consume the gain. section 6.3 section 6.3.3
