The 2017 Transformer was a turning point in machine translation. Researchers at Google replaced recurrent and convolutional layers with an architecture based on attention. It was not the first model to use attention; its breakthrough was making attention the central mechanism while outperforming earlier approaches on English-to-German and English-to-French translation benchmarks. section 5.0
A recurrent model processes a sequence token by token. In a Transformer, self-attention lets each position connect directly to other positions at each layer. A word such as “cheese” can therefore attend to “mice” without passing through every intervening word. The architecture combines this mechanism with multihead attention and positional encodings; its original design includes encoder and decoder stacks, feed-forward networks, and embeddings. section 5.1.1 section 5.0 section 5.2
What changed in practice
The “big” Transformer scored 28.4 BLEU on the WMT English-to-German benchmark, surpassing previous approaches. On WMT English-to-French, it scored 41.0 BLEU. Training also benefited from processing sequences without token-by-token recurrence: the “base” model took 12 hours on eight GPUs, while the “big” model took about 3.5 days. section 5.1.1
The design proved consequential beyond translation. GPT-2 was a transformer-based language model scaled up from GPT-1; its ability to produce extended, plausible text helped provoke debate over release and misuse. Larger Transformers also posed an engineering problem: GPipe partitioned one across 128 TPU devices to train an 83.9-billion-parameter model. Attention made a scalable architecture possible, but neither convincing output nor scale alone settles whether a model understands what it says. section 1.2 section 6.3.2 section 5.8
