chapter twelve
12 Beyond recurrence: Google Brain and the transformer architecture
This chapter covers
- Ashish Vaswani et al.’s Attention Is All You Need (2017) and the break from sequential state propagation toward relational modeling
- How self-attention overcomes recurrent constraints by preserving distant relationships and enabling parallel computation
- How queries, keys, values, softmax, and masking construct context and support token prediction
- How the transformer scales from local prediction to coherent generation across extended sequences
- Why the synthesis of probability, information theory, and generalization positions the transformer architecture for lasting influence
In 2015, the so-called godfathers of deep learning—Yann LeCun, Yoshua Bengio, and Geoffrey Hinton—argued that neural networks had effectively “arrived.” No longer academic curiosities, these layered models powered speech recognition, image classification, and early language systems at scale, learning representations directly from raw data and replacing manual feature design with end-to-end optimization.
Yet arrival did not imply culmination. Neural networks remained shaped by architectural assumptions, especially about how sequences should be processed. Language, speech, and time-series models typically processed input one step at a time, carrying prior context forward through recurrent neural networks (RNNs).