Parallelism lets multiple computations proceed concurrently across processing resources. In Transformer training, coordinated matrix operations can process many sequence tokens at once, unlike recurrent models, which propagate state sequentially. Autoregressive generation remains sequential: the model produces one token at a time. section 12.1.2 section 12.0
Adding processors does not guarantee proportionally faster computation:
- Memory limits, bandwidth, and communication or synchronization delays can leave processors waiting. section 13.5.1
- Coordination grows more difficult as systems expand, so greater parallelism has diminishing returns. section 13.5.1
- Electricity, cooling, networking, and construction also constrain the infrastructure that sustains computation. section 13.5.1
Training and generation
A Transformer can calculate relationships across many tokens concurrently during training, yet generate an autoregressive sequence one token at a time. section 12.1.2
Figure 12.1 The transformer architecture introduced in Attention Is All You Need. Stacked encoder and decoder layers combine multi-head attention with feed-forward transformations, residual connections, and normalization. Positional encodings preserve sequence order without recurrence. The decoder uses masked self-attention to prevent access to future tokens and cross-attention to incorporate encoder outputs. Within each multi-head attention block, individual heads use dot products to calculate compatibility scores and softmax to convert those scores into attention weights, as demonstrated in the worked examples that follow. Reproduced from Vaswani et al (2017), Attention Is All You Need, Proceedings of the 31st Conference on Neural Information Processing Systems (NeurIPS 2017).

The architecture diagram shows how attention and feed-forward operations fit within stacked Transformer layers. section 12.1.2






















































