chapter five

5 Attention is all you need

 

This chapter covers

  • How the Transformer replaced recurrence and convolution with self-attention.
  • Q/K/V, multi-head attention, and positional encodings.
  • seq2seq, “thought vectors,” the reversal trick, and “Order Matters.”
  • Cultural impact of the Transformer and the “X is All You Need” meme.
  • The Universal Transformer.
Papers
  • Attention is All You Need (2017) Vaswani, et al.
  • The Annotated Transformer (2018) Alexander Rush
  • Neural Machine Translation by Jointly Learning to Align and Translate (2014) Bahdanau, et al.
  • Pointer Networks (2015) Vinyals, et al.
  • Neural Turing Machines (2014) Graves, et al.
  • Order Matters: Sequence to Sequence for Sets (2015) Vinyals, et al.

In 2017, researchers at Google replaced recurrent and convolutional layers with a new architecture based on attention, which they called the Transformer.[1] The authors wrote, “We show that the Transformer outperforms both recurrent and convolutional models on academic English-to-German and English-to-French translation benchmarks.”[2] Beneath this measured academic tone lay the start of a revolution.

The Transformer was not the first model to use attention, but it nearly perfected the mechanism by synthesizing earlier innovations, including Bahdanau attention (2014), Graves attention (2014), and Luong’s refinements (2015).[3][4][5] This fusion was a catalytic moment that led to architectures such as the GPT family of models and, ultimately, ChatGPT.

5.1 Transformer

5.1.1 Empirical Results

5.1.2 Universal Transformer

5.2 The Annotated Transformer

5.3 Bahdanau Attention

5.4 Just Point to It!

5.4.1 Experimental Results

5.4.2 Significance

5.5 Neural Turing Machines

5.5.1 Significance

5.6 Order Matters

5.7 Reversing Input Sentences

5.8 Impact