liveBook · Manning

Parallelism

As explained in Machines that Think

Parallel computation is the simultaneous execution of multiple computations rather than processing them strictly one after another.

Parallelism is the simultaneous execution of multiple computations rather than processing them strictly one after another. In large language models, it helps make training at scale practical, provided that the architecture and hardware can use it efficiently. section 4.2.3 section 4.6.1

Recurrent models such as LSTMs and GRUs process language one token at a time, updating a state at each step. This sequence limits parallelism. Transformers abandon recurrence: attention allows tokens to interact directly, so entire sequences can be processed in parallel rather than threaded through the model position by position. section 3.5.3 section 4.2.3 section 4.1.1

Attention and feedforward layers express much of their workload as large matrix operations, which align with GPUs and TPUs. Parallel computation alone does not guarantee scalable training: signal flow must also remain stable, while memory and hardware coordination can become limiting factors. section 4.2.3

At larger scales, parallelism also involves distributing training across many accelerators. Parameters are split across devices, gradients are aggregated across machines, and activations are stored or recomputed to fit memory limits. Networking bottlenecks, idle devices, and unbalanced workloads can reduce utilization and increase cost. section 4.6.1

From sequential models to BERT

Early recurrent language models were constrained by slow sequential training. Parallel processing across whole sequences helped make models with hundreds of millions of parameters, such as BERT, practical to train. Parameter count alone, however, does not determine capability. section 4.2.1

How Machines that Think covers it

How deep it goes
Explains sequence-level and distributed parallelism and their constraints, without detailing specific parallelization algorithms.
Examples
Contrasts sequential LSTMs and GRUs with Transformers, and cites BERT as an example of training at larger scale.
What you can do after
Evaluate how architecture, hardware utilization, memory, and coordination affect the practicality of large-scale training.
How it connects
Connects parallelism to recurrence, attention, Transformers, distributed training, and model scaling.

Also explained in

Compare how 55 books explain parallelism →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.