Parallelism is the simultaneous execution of multiple computations rather than processing them strictly one after another. In large language models, it helps make training at scale practical, provided that the architecture and hardware can use it efficiently. section 4.2.3 section 4.6.1
Recurrent models such as LSTMs and GRUs process language one token at a time, updating a state at each step. This sequence limits parallelism. Transformers abandon recurrence: attention allows tokens to interact directly, so entire sequences can be processed in parallel rather than threaded through the model position by position. section 3.5.3 section 4.2.3 section 4.1.1
Attention and feedforward layers express much of their workload as large matrix operations, which align with GPUs and TPUs. Parallel computation alone does not guarantee scalable training: signal flow must also remain stable, while memory and hardware coordination can become limiting factors. section 4.2.3
At larger scales, parallelism also involves distributing training across many accelerators. Parameters are split across devices, gradients are aggregated across machines, and activations are stored or recomputed to fit memory limits. Networking bottlenecks, idle devices, and unbalanced workloads can reduce utilization and increase cost. section 4.6.1
From sequential models to BERT
Early recurrent language models were constrained by slow sequential training. Parallel processing across whole sequences helped make models with hundreds of millions of parameters, such as BERT, practical to train. Parameter count alone, however, does not determine capability. section 4.2.1






















































