liveBook · Manning

Chapter 12: Beyond recurrence: Google Brain and the transformer architecture

What you will learn in chapter 12 of Timeless Algorithms, The Foundational Ideas

The introduction explains why recurrent sequence models became a bottleneck and how the transformer’s attention-based architecture changed sequence representation and scaling.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction explains why recurrent sequence models became a bottleneck and how the transformer’s attention-based architecture changed sequence representation and scaling.

Key terms: recurrent neural networks (RNNs), recurrence, sequential, parallelism, transformer, attention, relational representation, parallel computation, large language models (LLMs)

12.1 From sequential processing to relational representation

This section introduces the transformer as a change in how a neural network represents and moves context, not a change in how it learns, and explains why the architecture suited large-scale research.

Key terms: forward pass, recurrent state propagation, self-attention, sequential compression, relational weighting, Google Brain, parallelizable matrix operations, GPUs, Tensor Processing Units (TPUs), large-scale experimentation

12.1.1 The constraints of sequential modeling

This section explains how recurrent models carry context through ordered sequences and why that design creates representational and computational constraints.

Key terms: recurrent neural networks, hidden state, context, fixed size, long short-term memory, gated recurrent units, recurrence, parallel hardware

12.1.2 From recurrence to self-attention

This section explains how transformers replace recurrent context with self-attention, supported by other architectural components, to make context construction more direct and training more parallel.

Key terms: self-attention, relational attention, hidden state, context-sensitive representation, positional information, multiple attention heads, feed-forward networks, residual connections, normalization, stacked layers

12.1.3 From representation to capability

This subsection explains how the transformer’s relational architecture provides a scalable foundation for capability, while other design and training choices shape the results.

Key terms: relational architecture, distant context, parallel training, model size, training data, objectives

12.2 Worked examples of attention in action

Introduces three attention examples and explains why their simplified calculations should not be mistaken for the behavior of a complete transformer.

Key terms: simplified vectors, attention heads, feed-forward networks, stacked layers

12.2.1 Example 1: resolving a pronoun through attention

A simplified pronoun example traces how self-attention turns tokens into contextual representations and next-token probabilities, while distinguishing this process from recurrence and from factual retrieval.

Key terms: tokenization, token identifiers, embedding matrix, embeddings, queries, keys, values, learned projection matrices, dot product, scaled compatibility scores

12.2.2 Semantic association with factual recall

The section uses an author-prediction example to show how a transformer turns a prompt and learned semantic associations into a likely factual continuation.

Key terms: semantic associations, backpropagation, learned parameters, final contextual representation, output layer, output logit, softmax, attention weights, database lookup, base language model

12.2.3 Example 3: autoregressive generation

The train example shows how causal masking and token-by-token prediction let a transformer build a multistep response, while also showing why a fluent solution can be wrong.

Key terms: causal masking, compatibility logits, softmax, attention weight, training, generation, recurrent models, autoregressive generation, expanded context, head start

12.3 Intellectual positioning

The section positions the transformer as an architectural change built on existing neural-network learning methods and distinguishes direct training foundations from broader interpretive connections.

Key terms: architecture, forward computation, backpropagation, gradient-based optimization, probabilistic prediction, likelihood-based estimation, information-based loss, Bayesian reasoning, statistical learning theory, MacKay’s synthesis

12.3.1 Foundations carried forward

This section explains which established learning principles directly govern transformer training and which provide broader ways to interpret its predictions and generalization.

Key terms: forward pass, loss function, backpropagation, self-attention, representation learning, next token, conditional distribution, context, softmax, vocabulary logits

12.3.2 Foundations reconsidered

This section explains how transformers model ordered language without requiring recurrent state propagation, a single fixed-state context, or sequential processing of training tokens—and where those distinctions have limits.

Key terms: recurrent state propagation, self-attention, positional encodings, Bellman’s recurrence relations, RNN, hidden state, fixed-state compression, representational bottleneck, token representations, attention heads

12.3.3 What was genuinely new

This section explains that the transformer’s innovation was an architecture that made attention central, combined multiple relational perspectives, and scaled through repeatable, parallel-friendly blocks rather than inventing new learning principles.

Key terms: recurrence, self-attention, contextual representations, direct relational modeling, multi-head attention, heads, query, key, and value projections, relational representation, repeated blocks, feed-forward transformations

12.4 The transformer’s prospects for lasting influence

This section weighs why transformers may remain influential without assuming their current architectural dominance will last.

Key terms: scalability, adaptability, technological ecosystem, computational costs, reliability limitations, direct relational modeling

12.4.1 Why the transformer is positioned to persist

The section explains why hardware fit, demonstrated performance, adaptability, and accumulated investment make transformers likely to persist without guaranteeing their dominance.

Key terms: matrix-based operations, GPUs and TPUs, training data, parameters, computation, transformer blocks, encoder-only, decoder-only, encoder–decoder, multimodal models

12.4.2 Why continued dominance is not assured

This section explains why the transformer’s computational costs, environmental impact, and reliability limits leave room for more efficient alternatives and hybrid architectures.

Key terms: standard self-attention, pairwise relationships, autoregressive generation, latency, inference cost, electricity, data-center infrastructure, model efficiency, hardware utilization, context window

12.5 Conclusion and transition

The section concludes that transformers reorganized established learning ideas around self-attention, shifted rather than eliminated constraints, and enabled the study of scaling laws.

Key terms: probability, likelihood, cross-entropy, backpropagation, gradient-based optimization, generalization, self-attention, recurrent compression, sequential training, finite context

12.6 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 11: Learning representations: LeCun, Bengio, Hinton, and the rise of deep learning Chapter 13: Expanding the boundaries of learning: OpenAI and the scaling laws of neural language models →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.