Parallelism computes multiple sequence elements simultaneously instead of processing one element at a time. In Transformer training, this reduces training time and supports large datasets. section 1.1.1 section 1.3
Compare parallelism with sequential processing:
- RNNs and LSTMs process sequence elements sequentially, limiting training speed and preventing efficient training on huge datasets. section 1.3
- Transformers process input sequences in parallel and use self-attention to capture relationships across long distances. section 9.0
- Parallel processing does not automatically preserve sequence order, so Transformers add positional encodings to input embeddings. section 10.5
Translation with a Transformer
For English-to-French translation, a Transformer processes the input sentence in parallel, while its decoder generates French tokens autoregressively from BOS until EOS. section 10.5
