liveBook · Manning

Parallelism

As explained in Timeless Algorithms, The Foundational Ideas

Parallel computation executes multiple computations concurrently across processing resources to reduce execution time or increase throughput.

Parallelism lets multiple computations proceed concurrently across processing resources. In Transformer training, coordinated matrix operations can process many sequence tokens at once, unlike recurrent models, which propagate state sequentially. Autoregressive generation remains sequential: the model produces one token at a time. section 12.1.2 section 12.0

Adding processors does not guarantee proportionally faster computation:

  • Memory limits, bandwidth, and communication or synchronization delays can leave processors waiting. section 13.5.1
  • Coordination grows more difficult as systems expand, so greater parallelism has diminishing returns. section 13.5.1
  • Electricity, cooling, networking, and construction also constrain the infrastructure that sustains computation. section 13.5.1

Training and generation

A Transformer can calculate relationships across many tokens concurrently during training, yet generate an autoregressive sequence one token at a time. section 12.1.2

Figure 12.1 The transformer architecture introduced in Attention Is All You Need. Stacked encoder and decoder layers combine multi-head attention with feed-forward transformations, residual connections, and normalization. Positional encodings preserve sequence order without recurrence. The decoder uses masked self-attention to prevent access to future tokens and cross-attention to incorporate encoder outputs. Within each multi-head attention block, individual heads use dot products to calculate compatibility scores and softmax to convert those scores into attention weights, as demonstrated in the worked examples that follow. Reproduced from Vaswani et al (2017), Attention Is All You Need, Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS 2017).

The architecture diagram shows how attention and feed-forward operations fit within stacked Transformer layers. section 12.1.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Distinguishes concurrent training from sequential generation, then identifies system bottlenecks and diminishing returns without detailing implementation.
Examples
Contrasts parallel token processing during training with one-token-at-a-time generation and distributed computation across machines.
What you can do after
Evaluate why adding processors may not reduce elapsed time when memory, communication, coordination, or infrastructure constrain computation.
How it connects
Links parallel computation to Transformer architecture, self-attention, recurrent neural networks, and distributed training.

Also explained in

Compare how 55 books explain parallelism →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.