Chapter 12: Beyond recurrence: Google Brain and the transformer architecture
The introduction explains why recurrent sequence models became a bottleneck and how the transformer’s attention-based architecture changed sequence representation and scaling.
After this chapter you will be able to
- Limits of Recurrent Sequence Modeling. Explain why processing a sequence one step at a time can make distant relationships harder to preserve and restrict parallel training.
- Transformer Shift to Relational Representation. Describe how the transformer departed from recurrence and why that architectural shift mattered for scaling and language-model capabilities.
- Learning Framework and Information Flow. Distinguish the learning framework the transformer retained from the information-flow structure it changed. Explain that prediction, error evaluation, backpropagation, and gradient-based updates remain, while self-attention replaces recurrent state propagation.
- Long-distance context compression. Explain why relationships between distant parts of a sequence can be difficult for a recurrent model to preserve, while distinguishing this tendency from an absolute inability.
- Sequential computation dependency. Explain why recurrence limits parallel processing during training.
- Self-attention and relational context. Explain how self-attention builds a context-sensitive representation for each token from weighted relationships with other tokens in the sequence, rather than carrying context through a recurrent hidden state.
- Parallel training and generation. Distinguish the transformer’s parallel processing during training from its one-token-at-a-time autoregressive generation, and explain the resulting architectural advantage without attributing all language-model capabilities to it.
- Sources of transformer capability. Explain why the transformer’s capabilities do not come from attention alone: its relational architecture gives direct access to distant context and supports parallel training, while model size, training data, objectives, and decoding methods affect how well those advantages are realized.
- Illustrative versus complete transformers. Explain why the examples use simplified vectors and why a complete model’s capabilities cannot be attributed to one attention calculation.
- Queries keys and compatibility. Explain the distinct roles of queries, keys, and values, and trace how the example scores the compatibility of it with trophy and box.
- Attention weights and contextualization. Trace how softmax turns compatibility scores into attention weights and how those weights combine value vectors for it.
- Context to token continuation. Trace how a contextual relationship can affect the probability of the token after too, and distinguish attention weights from output probabilities and token selection.
- Direct attention versus recurrence. Distinguish how a recurrent model and a transformer bring earlier context into a prediction, including the limit of the transformer’s advantage.
- Distributed title–author association. Explain how training can strengthen the association between The Count of Monte Cristo and Alexandre Dumas across many contexts, without storing the fact in one identifiable parameter.
- Contextual representation and output scoring. Trace how the prompt’s final contextual representation leads to an author prediction, and distinguish output-token scoring from attention over context tokens.
- Causal masking of future tokens. Explain how causal masking limits each position to earlier tokens and itself, and why that restriction matters when training on a complete sequence.
- Parallel training versus sequential generation. Distinguish the transformer's parallel calculation of masked positions during training from its sequential production of new tokens during generation.
- Chained train problem solution. Trace how the model appends each predicted token to its context and how the train example's intermediate statements support the final answer.
- Attention and error propagation limits. Explain attention's role in connecting prompt details and generated results, and why an early mistake can lead to a fluent but incorrect answer.
- Architecture versus learning theory. Explain that the transformer changed how information is represented and processed within a network without replacing the basic approach to neural-network training.
- Direct and interpretive foundations. Distinguish earlier ideas that operate directly in transformer training from ideas used to interpret the transformer but not built into its architecture.
- Inherited neural learning cycle. Explain how a transformer retains the neural-network cycle of prediction, loss measurement, backpropagation, and parameter updates despite changing how information moves through the network.
- Likelihood and information cost. Explain how likelihood and information theory give complementary interpretations of cross-entropy training for next-token prediction.
- Generalization beyond training data. Explain why low transformer training loss does not establish reliable performance on unseen data and how Vapnik’s framework applies without being built into attention.
- Sequence without recurrent propagation. Explain how self-attention and positional encodings let a transformer represent relationships and order without passing a hidden state from token to token. State that recurrence remains an architectural choice, not a universal requirement or a universally obsolete design.
- Distributed context and remaining limits. Describe how transformer context differs from an RNN’s evolving hidden state, why distant tokens become more directly accessible, and why compression and memory limits remain.
- Parallel training versus sequential generation. Explain why transformers can process many training positions in parallel while preserving next-token prediction, and why autoregressive generation still proceeds token by token.
- Self-attention as organizing operation. Distinguish the transformer’s use of self-attention from earlier attention added to recurrent models, and explain what removing recurrence changed about how context was represented.
- Multiple relational attention heads. Explain how multi-head attention broadens a token’s contextual representation and how stacking attention across layers can combine relationships.
- Modular parallel scaling. Describe how repeated transformer blocks provide ways to increase capacity and why their operations suit parallel training.
- Prospects for lasting influence. Explain why the transformer’s current prominence does not settle its long-term position, and describe the forces that support or challenge its continued influence.
- Scalability and demonstrated performance. Explain how transformers’ matrix-based operations suit parallel hardware and why observed gains from additional data, parameters, and computation encourage continued investment. Qualify those gains as neither unlimited nor guaranteed for every task.
- Reusable transformer configurations. Describe how transformer blocks can be rearranged for different purposes and applied to different numerically encoded inputs, making new applications less dependent on an entirely new framework.
- Ecosystem and replacement threshold. Explain how accumulated hardware, software, models, workflows, and expertise reinforce transformer use, and why a successor must offer enough practical benefit to justify migration.
- Attention and generation costs. Explain why longer contexts and generated responses can be expensive even though transformers scale well. Describe the quadratic growth of standard self-attention calculations and the sequential dependency in autoregressive generation.
- Context and reliability limits. Explain why a larger context window does not guarantee effective use of information, and why transformer outputs can be hard to explain or factually unreliable, especially in multistep tasks.
- Alternatives and hybrid futures. Distinguish the approaches described for reducing transformer costs, and explain why hybrid evolution is plausible without assuming that a replacement has already succeeded.
- Foundations within transformer architecture. Explain that the transformer reorganized established foundations rather than replacing them, including probability, likelihood or cross-entropy, neural-network optimization, and generalization.
- Relational access and remaining limits. Describe how self-attention made distant and overlapping relationships more directly available than recurrent updates, while shifting rather than eliminating architectural constraints.
What it builds on
- Architecture and representation (chapter 11) · Introduces neural-network architecture and representation.
- The complete learning cycle (chapter 11) · The neural-network cycle of prediction, error evaluation, and parameter updates.
- Updating the network (chapter 11) · Training adjusts learned parameters in response to prediction error.
- Making the initial prediction (chapter 11) · Numerical scores can be transformed for use in a prediction calculation.
- From input to prediction (chapter 11) · A neural network transforms internal representations into prediction scores.
- Tracing error backward (chapter 11) · Backpropagation of prediction error through a neural network
- Synthesis rather than invention (chapter 10) · MacKay’s role in synthesizing earlier learning ideas.
- Cross-entropy as likelihood and information (chapter 11) · probabilistic predictions and cross-entropy in neural-network learning
Section by section
Introduction
The introduction explains why recurrent sequence models became a bottleneck and how the transformer’s attention-based architecture changed sequence representation and scaling.
- Limits of Recurrent Sequence Modeling: Explain why processing a sequence one step at a time can make distant relationships harder to preserve and restrict parallel training.
Why it matters: These limitations explain why a different sequence architecture was valuable. - Transformer Shift to Relational Representation: Describe how the transformer departed from recurrence and why that architectural shift mattered for scaling and language-model capabilities.
Why it matters: The change was about how models represent sequence structure, not just faster training.
Key terms: recurrent neural networks (RNNs), recurrence, sequential, parallelism, transformer, attention, relational representation, parallel computation, large language models (LLMs)
12.1 From sequential processing to relational representation
This section introduces the transformer as a change in how a neural network represents and moves context, not a change in how it learns, and explains why the architecture suited large-scale research.
- Learning Framework and Information Flow: Distinguish the learning framework the transformer retained from the information-flow structure it changed. Explain that prediction, error evaluation, backpropagation, and gradient-based updates remain, while self-attention replaces recurrent state propagation.
Why it matters: This distinction keeps the architectural innovation separate from the learning process shared with earlier neural networks. - Conditions for Transformer Scaling: Explain how Google Brain's research environment and the transformer's parallelizable operations supported efficient training at greater scale. State how the section connects that combination to the transformer's later influence.
Why it matters: The section presents the transformer's reach as a result of architecture working together with computing resources and experimentation.
Key terms: forward pass, recurrent state propagation, self-attention, sequential compression, relational weighting, Google Brain, parallelizable matrix operations, GPUs, Tensor Processing Units (TPUs), large-scale experimentation
12.1.1 The constraints of sequential modeling
This section explains how recurrent models carry context through ordered sequences and why that design creates representational and computational constraints.
- Context in recurrent hidden states: Describe how a recurrent neural network processes an ordered sequence and uses its hidden state to carry earlier context into later predictions.
Why it matters: This explains both how recurrence uses context and where its constraints originate. - Long-distance context compression: Explain why relationships between distant parts of a sequence can be difficult for a recurrent model to preserve, while distinguishing this tendency from an absolute inability.
Why it matters: It identifies the representational pressure that grows as sequences and relationships become more complex. - Sequential computation dependency: Explain why recurrence limits parallel processing during training.
Why it matters: This explains why sequential processing became restrictive as demands for scale grew.
Key terms: recurrent neural networks, hidden state, context, fixed size, long short-term memory, gated recurrent units, recurrence, parallel hardware
12.1.2 From recurrence to self-attention
This section explains how transformers replace recurrent context with self-attention, supported by other architectural components, to make context construction more direct and training more parallel.
- Self-attention and relational context: Explain how self-attention builds a context-sensitive representation for each token from weighted relationships with other tokens in the sequence, rather than carrying context through a recurrent hidden state.
Why it matters: This is the central change in how the transformer represents context. - Supporting transformer components: Describe how positional information, multiple attention heads, feed-forward networks, residual connections, and normalization support self-attention in stacked transformer layers.
Why it matters: These components explain why the architecture is more than attention alone. - Parallel training and generation: Distinguish the transformer’s parallel processing during training from its one-token-at-a-time autoregressive generation, and explain the resulting architectural advantage without attributing all language-model capabilities to it.
Why it matters: It separates the transformer’s architectural contribution from both sequential generation and other causes of model capability.
Key terms: self-attention, relational attention, hidden state, context-sensitive representation, positional information, multiple attention heads, feed-forward networks, residual connections, normalization, stacked layers
12.1.3 From representation to capability
This subsection explains how the transformer’s relational architecture provides a scalable foundation for capability, while other design and training choices shape the results.
- Sources of transformer capability: Explain why the transformer’s capabilities do not come from attention alone: its relational architecture gives direct access to distant context and supports parallel training, while model size, training data, objectives, and decoding methods affect how well those advantages are realized.
Why it matters: It separates what the architecture makes possible from the other factors that shape what a transformer can do.
Key terms: relational architecture, distant context, parallel training, model size, training data, objectives
12.2 Worked examples of attention in action
Introduces three attention examples and explains why their simplified calculations should not be mistaken for the behavior of a complete transformer.
- Illustrative versus complete transformers: Explain why the examples use simplified vectors and why a complete model’s capabilities cannot be attributed to one attention calculation.
Why it matters: This distinction helps you interpret the worked examples without overclaiming what one calculation explains.
Key terms: simplified vectors, attention heads, feed-forward networks, stacked layers
12.2.1 Example 1: resolving a pronoun through attention
A simplified pronoun example traces how self-attention turns tokens into contextual representations and next-token probabilities, while distinguishing this process from recurrence and from factual retrieval.
- Tokens and learned embeddings: Describe how text becomes token identifiers and then numerical embeddings, and explain why an identifier’s number is not its meaning.
Why it matters: These vectors are the starting point for the attention calculation. - Queries keys and compatibility: Explain the distinct roles of queries, keys, and values, and trace how the example scores the compatibility of it with trophy and box.
Why it matters: Compatibility scores show how attention identifies which context tokens may matter more. - Attention weights and contextualization: Trace how softmax turns compatibility scores into attention weights and how those weights combine value vectors for it.
Why it matters: Attention constructs useful context rather than issuing a separate, certain pronoun-resolution decision. - Context to token continuation: Trace how a contextual relationship can affect the probability of the token after too, and distinguish attention weights from output probabilities and token selection.
Why it matters: The model predicts a next token from completed context, not directly from a pronoun’s attention weight. - Direct attention versus recurrence: Distinguish how a recurrent model and a transformer bring earlier context into a prediction, including the limit of the transformer’s advantage.
Why it matters: The architectural distinction concerns how predictive context is built, not whether either model can output next-token probabilities. - Generation memory and retrieval: Explain how token-by-token generation relates to learned patterns, possible memorization, external retrieval, and factual reliability.
Why it matters: It prevents fluent generation from being mistaken for a verified lookup or a guarantee of novelty.
Key terms: tokenization, token identifiers, embedding matrix, embeddings, queries, keys, values, learned projection matrices, dot product, scaled compatibility scores
12.2.2 Semantic association with factual recall
The section uses an author-prediction example to show how a transformer turns a prompt and learned semantic associations into a likely factual continuation.
- Distributed title–author association: Explain how training can strengthen the association between The Count of Monte Cristo and Alexandre Dumas across many contexts, without storing the fact in one identifiable parameter.
Why it matters: It explains how a model can answer a question whose answer is not stated in the prompt. - Contextual representation and output scoring: Trace how the prompt’s final contextual representation leads to an author prediction, and distinguish output-token scoring from attention over context tokens.
Why it matters: It separates what attention does with the prompt from how the model chooses a possible answer. - Factual reconstruction and fallibility: Distinguish a base language model’s factual answer from a database lookup and explain why a plausible answer may still be wrong.
Why it matters: A fluent factual answer is a prediction, not a guarantee that the fact was verified.
Key terms: semantic associations, backpropagation, learned parameters, final contextual representation, output layer, output logit, softmax, attention weights, database lookup, base language model
12.2.3 Example 3: autoregressive generation
The train example shows how causal masking and token-by-token prediction let a transformer build a multistep response, while also showing why a fluent solution can be wrong.
- Causal masking of future tokens: Explain how causal masking limits each position to earlier tokens and itself, and why that restriction matters when training on a complete sequence.
Why it matters: The mask makes next-token learning possible without letting the model peek ahead. - Parallel training versus sequential generation: Distinguish the transformer's parallel calculation of masked positions during training from its sequential production of new tokens during generation.
Why it matters: The transformer can parallelize training positions without producing an entire new response at once. - Chained train problem solution: Trace how the model appends each predicted token to its context and how the train example's intermediate statements support the final answer.
Why it matters: Each generated intermediate result becomes context for what the model says next. - Attention and error propagation limits: Explain attention's role in connecting prompt details and generated results, and why an early mistake can lead to a fluent but incorrect answer.
Why it matters: A convincing chain of generated steps should not be mistaken for a guaranteed correct calculation.
Key terms: causal masking, compatibility logits, softmax, attention weight, training, generation, recurrent models, autoregressive generation, expanded context, head start
12.3 Intellectual positioning
The section positions the transformer as an architectural change built on existing neural-network learning methods and distinguishes direct training foundations from broader interpretive connections.
- Architecture versus learning theory: Explain that the transformer changed how information is represented and processed within a network without replacing the basic approach to neural-network training.
Why it matters: This distinction clarifies what the transformer changed and what it inherited. - Direct and interpretive foundations: Distinguish earlier ideas that operate directly in transformer training from ideas used to interpret the transformer but not built into its architecture.
Why it matters: It prevents a useful intellectual connection from being mistaken for a component of the architecture.
Key terms: architecture, forward computation, backpropagation, gradient-based optimization, probabilistic prediction, likelihood-based estimation, information-based loss, Bayesian reasoning, statistical learning theory, MacKay’s synthesis
12.3.1 Foundations carried forward
This section explains which established learning principles directly govern transformer training and which provide broader ways to interpret its predictions and generalization.
- Inherited neural learning cycle: Explain how a transformer retains the neural-network cycle of prediction, loss measurement, backpropagation, and parameter updates despite changing how information moves through the network.
Why it matters: The transformer changes representation and information flow without replacing the basic neural-network learning process. - Conditional next-token predictions: Describe what a transformer-based language model predicts from preceding tokens and how its logits become probabilities.
Why it matters: Next-token probabilities are the predictions to which the section’s training and information-theory explanations apply. - Likelihood and information cost: Explain how likelihood and information theory give complementary interpretations of cross-entropy training for next-token prediction.
Why it matters: These two views explain the same training objective without treating the transformer’s new architecture as a new estimation principle. - Limits of Bayesian inheritance: Distinguish the broad connection to Bayesian conditional reasoning from what a standard transformer ordinarily does during prediction.
Why it matters: It prevents a broad intellectual connection from being mistaken for a mechanism inside the model. - Generalization beyond training data: Explain why low transformer training loss does not establish reliable performance on unseen data and how Vapnik’s framework applies without being built into attention.
Why it matters: Good fit to training examples is not by itself evidence that a transformer will work reliably elsewhere. - MacKay’s connected learning lens: Describe how MacKay’s synthesis connects prediction, information cost, and optimization in transformer learning, without treating it as a self-attention mechanism.
Why it matters: It ties the section’s inherited ideas together while keeping explanatory frameworks distinct from architectural components.
Key terms: forward pass, loss function, backpropagation, self-attention, representation learning, next token, conditional distribution, context, softmax, vocabulary logits
12.3.2 Foundations reconsidered
This section explains how transformers model ordered language without requiring recurrent state propagation, a single fixed-state context, or sequential processing of training tokens—and where those distinctions have limits.
- Sequence without recurrent propagation: Explain how self-attention and positional encodings let a transformer represent relationships and order without passing a hidden state from token to token. State that recurrence remains an architectural choice, not a universal requirement or a universally obsolete design.
Why it matters: It separates the order of language from one particular way of computing over that order. - Two meanings of recurrence: Distinguish an RNN’s repeated hidden-state updates from Bellman’s recurrence relations for decomposing sequential optimization problems.
Why it matters: It prevents a claim about transformer architecture from being mistaken for a claim about dynamic programming. - Distributed context and remaining limits: Describe how transformer context differs from an RNN’s evolving hidden state, why distant tokens become more directly accessible, and why compression and memory limits remain.
Why it matters: It explains both the benefit of direct access to distant context and the limits that attention does not remove. - Parallel training versus sequential generation: Explain why transformers can process many training positions in parallel while preserving next-token prediction, and why autoregressive generation still proceeds token by token.
Why it matters: It explains the transformer’s training-efficiency gain without confusing parallel training with parallel output generation.
Key terms: recurrent state propagation, self-attention, positional encodings, Bellman’s recurrence relations, RNN, hidden state, fixed-state compression, representational bottleneck, token representations, attention heads
12.3.3 What was genuinely new
This section explains that the transformer’s innovation was an architecture that made attention central, combined multiple relational perspectives, and scaled through repeatable, parallel-friendly blocks rather than inventing new learning principles.
- Self-attention as organizing operation: Distinguish the transformer’s use of self-attention from earlier attention added to recurrent models, and explain what removing recurrence changed about how context was represented.
Why it matters: This distinction identifies what was architecturally new without crediting the transformer with inventing attention. - Multiple relational attention heads: Explain how multi-head attention broadens a token’s contextual representation and how stacking attention across layers can combine relationships.
Why it matters: Multi-head attention helps explain how the architecture can preserve and combine several dependencies rather than relying on one pattern of relevance. - Modular parallel scaling: Describe how repeated transformer blocks provide ways to increase capacity and why their operations suit parallel training.
Why it matters: The architecture’s repeatability and hardware compatibility help explain why additional data, computation, and model capacity could be put to use. - Continuity and scaling limits: Distinguish the transformer’s architectural contribution from the learning foundations it retained, and state why its influence does not guarantee permanence.
Why it matters: This keeps the architectural advance in perspective: scaling enabled broad applications, but neither replaced learning fundamentals nor removed practical limits.
Key terms: recurrence, self-attention, contextual representations, direct relational modeling, multi-head attention, heads, query, key, and value projections, relational representation, repeated blocks, feed-forward transformations
12.4 The transformer’s prospects for lasting influence
This section weighs why transformers may remain influential without assuming their current architectural dominance will last.
- Prospects for lasting influence: Explain why the transformer’s current prominence does not settle its long-term position, and describe the forces that support or challenge its continued influence.
Why it matters: It separates the lasting influence of an idea from the lasting dominance of one architecture.
Key terms: scalability, adaptability, technological ecosystem, computational costs, reliability limitations, direct relational modeling
12.4.1 Why the transformer is positioned to persist
The section explains why hardware fit, demonstrated performance, adaptability, and accumulated investment make transformers likely to persist without guaranteeing their dominance.
- Scalability and demonstrated performance: Explain how transformers’ matrix-based operations suit parallel hardware and why observed gains from additional data, parameters, and computation encourage continued investment. Qualify those gains as neither unlimited nor guaranteed for every task.
Why it matters: Hardware compatibility and observed results help explain why investment in transformers continues. - Reusable transformer configurations: Describe how transformer blocks can be rearranged for different purposes and applied to different numerically encoded inputs, making new applications less dependent on an entirely new framework.
Why it matters: Reusing a framework can make it easier to try transformers in new domains. - Ecosystem and replacement threshold: Explain how accumulated hardware, software, models, workflows, and expertise reinforce transformer use, and why a successor must offer enough practical benefit to justify migration.
Why it matters: A model’s practical success depends partly on the tools and investments surrounding it, not just benchmark performance.
Key terms: matrix-based operations, GPUs and TPUs, training data, parameters, computation, transformer blocks, encoder-only, decoder-only, encoder–decoder, multimodal models
12.4.2 Why continued dominance is not assured
This section explains why the transformer’s computational costs, environmental impact, and reliability limits leave room for more efficient alternatives and hybrid architectures.
- Attention and generation costs: Explain why longer contexts and generated responses can be expensive even though transformers scale well. Describe the quadratic growth of standard self-attention calculations and the sequential dependency in autoregressive generation.
Why it matters: These costs help explain why success at scale does not guarantee continued dominance. - Environmental costs of scale: Describe the infrastructure and electricity demands of large transformer-based systems, and explain why efficiency improvements may not reduce their total impact.
Why it matters: Environmental impact depends on the scale of the whole system, not just the efficiency of one calculation. - Context and reliability limits: Explain why a larger context window does not guarantee effective use of information, and why transformer outputs can be hard to explain or factually unreliable, especially in multistep tasks.
Why it matters: Having relevant text available and producing fluent language are not guarantees of correct reasoning or factual answers. - Alternatives and hybrid futures: Distinguish the approaches described for reducing transformer costs, and explain why hybrid evolution is plausible without assuming that a replacement has already succeeded.
Why it matters: Architectural change may preserve what transformers do well while addressing costs they handle less well.
Key terms: standard self-attention, pairwise relationships, autoregressive generation, latency, inference cost, electricity, data-center infrastructure, model efficiency, hardware utilization, context window
12.5 Conclusion and transition
The section concludes that transformers reorganized established learning ideas around self-attention, shifted rather than eliminated constraints, and enabled the study of scaling laws.
- Foundations within transformer architecture: Explain that the transformer reorganized established foundations rather than replacing them, including probability, likelihood or cross-entropy, neural-network optimization, and generalization.
Why it matters: This distinguishes an architectural change from a wholly new theory of learning. - Relational access and remaining limits: Describe how self-attention made distant and overlapping relationships more directly available than recurrent updates, while shifting rather than eliminating architectural constraints.
Why it matters: The change explains both the transformer's expanded capabilities and why it is not a solution to every modeling problem. - Scaling laws as empirical regularities: Distinguish scaling laws from a new learning mechanism: they describe observed regularities in performance as architecture, data, parameters, and computation grow together.
Why it matters: This frames the next development as an observation about scaled systems, not a replacement for their learning foundations.
Key terms: probability, likelihood, cross-entropy, backpropagation, gradient-based optimization, generalization, self-attention, recurrent compression, sequential training, finite context
12.6 Summary
Where this is used later
- From learned representations to relational architecture (chapter 13) · The discussion of relational architecture builds on its contribution to scalable learning.
- The empirical puzzle (chapter 13) · Provides earlier context for predicting text from preceding context.
- Measuring performance across resources (chapter 13) · Introduces prediction of a following token from preceding tokens.
- Compute: the work of learning (chapter 13) · Provides context for the role of training compute.
- Computational and physical boundaries (chapter 13) · Provides context for computational boundaries as systems scale.
- The empirical pattern (chapter 13) · The observed performance gains provide context for the scaling pattern examined there.
Concepts explained in this chapter
- self-attentionSelf-attention forms representations of sequence elements by relating each element to others in the same sequence.
- autoregressive generationAutoregressive generation is the process of producing a sequence by predicting one token at a time, conditioning each prediction on tokens already available.
- transformer architectureA transformer is a neural-network architecture that uses attention to model relationships among sequence elements.
- logitA logit is an unnormalized real-valued score that can be transformed into a probability.
- attention weightAn attention weight is a normalized value indicating how strongly one element contributes to an attention-based representation.
- tokenA token is a discrete unit of text represented in a model's vocabulary, such as a word, word fragment, or punctuation mark.
- embeddingAn embedding is a numerical vector representation of an item, such as a word or token, in a learned space.
- inner productA dot product is an operation that maps two equal-length numerical vectors to the sum of their pairwise coordinate products.
- keyIn an attention mechanism, a projected representation of an element that is compared with queries to determine relevance.
- queryIn an attention mechanism, a projected representation of an element that is compared with keys to determine which information to attend to.
- tokenizationThe process of dividing text into discrete units called tokens.
- valueA projected representation of information that is combined with other values according to attention weights.
- masked attentionCausal masking is a restriction on attention that prevents a sequence position from accessing subsequent positions.
- attentionAn attention mechanism weights relationships among elements to form context-dependent representations.
- parallelismParallel computation executes multiple computations concurrently across processing resources to reduce execution time or increase throughput.
- recurrent neural networkA neural network that processes sequential inputs using a state carried across sequence positions.
- context windowThe finite amount of input context a model can consider when producing an output.
- multi-head attentionAn attention mechanism that computes multiple attention patterns in parallel using distinct learned projections and combines their outputs.
- hidden stateA hidden state is an internal representation that a recurrent neural network carries forward to summarize information from earlier sequence elements.
- semantic associationA semantic association is a learned relationship between concepts that makes one concept more likely to be recalled or predicted in the presence of another.
- contextual embeddingA contextual representation is a numerical encoding of a token or sequence that incorporates information from its surrounding context.
- backpropagationA backward pass propagates learning signals from a model’s output back through its components to determine their contributions to the loss.
- cross entropy lossCross-entropy loss measures the discrepancy between target outcomes and predicted probabilities.
- generalizationGeneralization is the ability of a learned model to perform effectively on observations not used during training.
- likelihoodA likelihood function measures how compatible observed data are with different parameter values of a statistical model.
- statistical learning theoryStatistical learning theory is a mathematical framework for analyzing how data, model capacity, and learning procedures affect predictive performance.
- scaling lawA scaling law describes how a system’s properties change as its scale or resource levels vary.
- attention headA component of an attention architecture that evaluates relationships among sequence elements, potentially focusing on a different type of relationship from other heads.
- decoding strategyA procedure for selecting a generated token from a model’s next-token probability distribution.
- feedforward networkA neural network component that applies learned transformations independently at each sequence position.
- token idAn integer index that identifies a token in a model’s vocabulary.
- next-word predictionA language-modeling objective in which a model learns to predict the next token from preceding tokens in a sequence.
- expressive powerThe range and complexity of relationships that a model can represent.
- recurrenceA rule that defines a quantity in terms of its values for other instances.
- softmaxThe softmax function transforms a vector of real-valued scores into a probability distribution.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
