Parallel processing executes computations across multiple parts of a sequence simultaneously. In Transformer models, self-attention processes an entire sentence at once instead of carrying information through one recurrent step at a time. section 2.1 section 2.0
Transformer parallelism provides several practical advantages:
- Speeds training and inference by removing recurrent, step-by-step processing. section 2.1
- Lets self-attention compare positions across the whole input, supporting long-range dependency handling. section 2.1
- Improves scalability for larger datasets and more complex tasks. section 2.1
Reading a paragraph at once
Reading a book word by word resembles an RNN or GRU: each step updates a hidden state. Reading a paragraph in a single glance resembles a Transformer: self-attention examines all words and their relationships together, including connections between distant words. section 2.1
Figure 2.1 The strength of the transformer model is in its ability to understand the significance and context of every word in a sentence, unlike RNNs which primarily assess the connection of a word with its immediate neighbor.
Use the figure to compare Transformer attention across a sentence with the more local word-to-word assessment associated with RNNs. section 2.1






















































