Megatron-LM slices transformer layers across GPUs: matrix multiplications are partitioned, and devices combine their partial results. This offers an alternative to pipeline parallelism, whose fill and drain phases can leave devices idle. The approaches are orthogonal and can be combined. section 6.3.3
Combining parallelism
Eight pipeline stages, each distributed in two ways, produce 16-way parallelism. Megatron-LM also trained an 8.3-billion-parameter GPT-2-style transformer on 512 GPUs, at about 76% efficiency and 15 petaflops of throughput. section 6.3.3
