Attention
Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
Inside Deep Learning: Math, Algorithms, Models
A technique that assigns weights to inputs and combines them so a model can focus on relevant information.
- How it is used
- Deploys learned relative importance when inputs contain multiple items or when only part of an input is relevant.
- How deep it goes
- Descends from intuition through equations, PyTorch modules, context scores, masking, variable lengths, and Seq2Seq integration.
- Examples
- Demonstrates largest-digit MNIST bags, variable-length inputs, machine translation, attention maps, and attention-weighted embeddings.
- What you can do after
- Build it from scratch in PyTorch, handle padded variable-length inputs, compare score functions, and inspect learned weights.
Read how Inside Deep Learning: Math, Algorithms, Models explains attention →
Learn Generative AI with PyTorch
A neural sequence-processing technique that assigns learned weights to relationships between elements, including self-relationships, to model relevance across a sequence.
- How it is used
- Connects distant sequence elements in Transformers used for translation, text generation, and music generation.
- How deep it goes
- Develops scaled dot-product attention from learned projections through softmax and weighted values, then implements causal masking and multihead tensor reshaping.
- Examples
- Works through a four-token sentence, a PyTorch causal mask, English-to-French translation, and musical-event sequences.
- What you can do after
- Build it from scratch in PyTorch for translation or decoder-only generation, including multihead causal self-attention.
Read how Learn Generative AI with PyTorch explains attention →
Sutskever's List
An attention mechanism computes weighted interactions between elements to focus a neural network computation on task-relevant information.
- How it is used
- Uses attention to bypass the fixed-context bottleneck in translation and connect distant positions in Transformers.
- How deep it goes
- Explains relevance scores, weights, and context vectors for Bahdanau attention; treats Transformer self-attention through direct connections and computational trade-offs rather than a derivation.
- Examples
- Illustrates attention with English–French word alignments, BLEU comparisons, and direct links between distant words.
- What you can do after
- Evaluate/understand when attention helps sequence models and why all-pairs self-attention becomes costly for long inputs.
Deep Learning with R, Third Edition
An attention layer computes context-dependent representations by allowing sequence elements to weight information from other elements.
- How it is used
- Uses attention to move relevant information across sequences and within Transformer encoders and decoders for translation and other language tasks.
- How deep it goes
- Works through dot-product scoring, softmax-weighted values, learned projections, multiple heads, masks, and Transformer blocks using R/Keras implementations rather than formal derivations.
- Examples
- Uses English-to-Spanish translation, a train-station sentence, R/Keras attention layers, and pretrained movie-review classification.
- What you can do after
- Build it from scratch with R/Keras dot-product attention, then apply layer_multi_head_attention in masked Transformer translation models.
Read how Deep Learning with R, Third Edition explains attention →
Build a Text-to-Image Generator (from Scratch)
A neural-network mechanism that weights input elements according to their relevance when computing a representation or prediction.
- How it is used
- Weights relevant words in translation and image regions in classification or denoising.
- How deep it goes
- Works through scaled dot-product scores, masking, softmax, and weighted values in code; introduces multi-head splitting without displaying its full class definition.
- Examples
- Uses German-to-English translation, a PyTorch attention function, image-patch classification, and high-resolution flower-image denoising.
- What you can do after
- Build it from scratch in an encoder–decoder transformer and apply the mechanism to image patches or a denoising U-Net.
Read how Build a Text-to-Image Generator (from Scratch) explains attention →
Deep Learning with Python, Third Edition
A mechanism that assigns context-dependent weights to parts of a sequence so a model can emphasize some elements over others when processing input.
- How it is used
- Uses context-dependent token weighting to move information across sequences for translation and within sequences for contextual representations.
- How deep it goes
- Works through dot-product scores, softmax-weighted sums, learned projections, multiple heads, encoder–decoder placement, and masks using pseudocode and Keras blocks.
- Examples
- Illustrates source-word weighting during translation, contextualizes “station” with “train,” and shows a lower-triangular decoder mask.
- What you can do after
- Build it from scratch with attention-based encoder and decoder blocks, then apply causal and padding masks for translation.
Read how Deep Learning with Python, Third Edition explains attention →
Transformers in Action
An attention mechanism dynamically assigns greater computational weight to input elements that are more relevant to the current representation or prediction.
- How it is used
- Uses attention to weight relevant sequence positions when transformers form contextual representations for tasks such as translation.
- How deep it goes
- Works from position-wise weights through scaled dot-product computation to simplified NumPy and PyTorch implementations.
- Examples
- Uses English-to-French translation, a five-position weight calculation, and simplified scaled-attention and multihead-attention listings.
- What you can do after
- Build it from scratch with NumPy projections and weighted values, then inspect a simplified PyTorch multihead implementation.
Build a Large Language Model (From Scratch)
A neural-network mechanism that weights input tokens or values by relevance so the model can focus on important context.
- How it is used
- Deploys attention inside transformer language models to select relevant context while generating or representing sequences.
- How deep it goes
- Explains weighted sums, dot products, queries, keys, values, causal masks, dropout, and multi-head stacking, without formal derivations.
- Examples
- Demonstrates attention through simplified dot products, weighted context vectors, causal masks, dropout masks, and stacked attention modules.
- What you can do after
- Build it from scratch as simplified, causal, and multi-head attention modules for a GPT-like language model.
Read how Build a Large Language Model (From Scratch) explains attention →
Grokking AI Algorithms, Second Edition
Attention is a neural-network mechanism that weights relationships among input representations so relevant information contributes more strongly to a computation.
- How it is used
- Uses attention to select relevant context for language tokens and connect prompt words with image features.
- How deep it goes
- Works through learned projections, scaled scores, softmax weights, and weighted context vectors for one attention head.
- Examples
- Traces a reading-highlighter analogy, an ingredient analogy, and a 32-token context-vector calculation.
- What you can do after
- Build it from scratch with a learning-oriented Python function that projects embeddings, calculates weights, and blends Values.
Read how Grokking AI Algorithms, Second Edition explains attention →
LLMs in Production
An attention mechanism dynamically weights relationships between elements of an input sequence according to their relevance to a current computation.
- How it is used
- Weights relevant sequence elements to support larger-context language modeling and attention-based transformer architectures.
- How deep it goes
- Works through query–key scoring and value weighting with a NumPy implementation, while deferring encoder–decoder sequence-length details.
- Examples
- Uses a NumPy multihead-attention listing and a GPT-3-to-GPT-4 upgrade that increased inference time and timeouts.
- What you can do after
- Build it from scratch with the illustrated NumPy operations and evaluate context-length increases against latency and memory constraints.
Natural Language Processing in Action, Second Edition
An attention mechanism is a sequence-processing operation that assigns different importance to input elements when computing a representation or output.
- How it is used
- Connects important patterns across long text sequences for transformer encoding and language generation.
- How deep it goes
- Explains sequence-wide matrix application, recurrence removal, fixed-length limits, and stackable encodings, but does not detail training.
- Examples
- Uses translation, long passages, token sequences such as “What is BERT?”, attention matrices, and stackable transformer diagrams.
- What you can do after
- Evaluate how transformers connect distant tokens while trading recurrence problems for fixed-length sequence constraints.
Read how Natural Language Processing in Action, Second Edition explains attention →
Deep Learning with PyTorch, Second Edition
A neural-network technique that helps a model focus on the most relevant parts of its input when computing representations.
- How it is used
- Uses attention to make sequence representations responsive to context during character-level generation and transformer text generation.
- How deep it goes
- Works through query, key, and value projections, softmax weights, and weighted outputs; leaves cross-attention as an exercise.
- Examples
- Demonstrates dot-product attention with PyTorch tensors and relates contextual focus to characters in names and image patches.
- What you can do after
- Build it from scratch using PyTorch linear projections, softmax, and weighted values in a small self-attention calculation.
Read how Deep Learning with PyTorch, Second Edition explains attention →
Graph Neural Networks in Action
An attention mechanism assigns learned, data-dependent importance weights to inputs when computing a representation.
- How it is used
- Highlights influential graph neighbors when node importance may not be apparent from degree.
- How deep it goes
- Defines attention intuitively, distinguishes related forms, and points to GAT equations for a deeper account.
- Examples
- Uses a nonlinear-story analogy and a fake-review detection network to illustrate selective emphasis.
- What you can do after
- Apply it via PyTorch Geometric GAT or GATv2 layers and compare against a GCN baseline.
Read how Graph Neural Networks in Action explains attention →
Grokking AI Applications
An attention mechanism computes the relative relevance of input elements so a model can focus on the most informative parts of a sequence.
- How it is used
- Explains how transformer-based language models identify relevant tokens while processing natural-language sequences.
- How deep it goes
- Defines the relevance-focused mechanism and gives one example, but stops short of its neural implementation details.
- Examples
- Uses the question about Italy’s capital to show why relevant tokens receive greater focus.
- What you can do after
- Evaluate how attention helps a language model select informative sequence elements and recognize context-window constraints.
Optimization Algorithms: AI techniques for design, planning, and control problems
An attention mechanism computes weights over representations so a model can focus on selected input information when producing an output.
- How it is used
- Uses learned relevance weights to focus sequence models or graph models on the input information needed for each output.
- How deep it goes
- Defines selective focus, scores, and applications, but stops short of deriving the attention computation or training procedure.
- Examples
- Illustrates attention with long-sequence translation, graph neighbors, convex hull selection, and the output sequence [1, 4, 2].
- What you can do after
- Evaluate how attention prioritizes sequence segments or graph neighbors and understand its use in variable-sized output selection.
Domain-Specific Small Language Models
A sequence-modeling technique that weights relevant parts of an input more heavily than less relevant parts.
- How it is used
- Explains sequence-model weighting and frames the performance costs of attention during Transformer inference.
- How deep it goes
- Defines attention and self-attention, then traces KV caching and arithmetic intensity without formal derivation.
- Examples
- Uses next-token weighting, cached Q/K/V computation, and GPT-Neo generation timing with and without KV caching.
- What you can do after
- Operate/configure Transformers generation with KV caching and diagnose attention performance using arithmetic intensity.
Read how Domain-Specific Small Language Models explains attention →
Machine Learning Algorithms in Depth
Attention is a technique that computes context-dependent weighted combinations of input representations according to their relevance to a query.
- How it is used
- Uses adaptive weights to give sequence outputs direct, context-dependent access to input representations.
- How deep it goes
- Derives query–key–value weighting and a matrix form, then connects them to a transformer classifier implemented with Keras/TensorFlow layers.
- Examples
- Illustrates attention with a neural-machine-translation heatmap and a Keras/TensorFlow transformer text classifier.
- What you can do after
- Apply it via Keras/TensorFlow in a transformer text classifier and interpret query–key weights as weighted access to values.
Read how Machine Learning Algorithms in Depth explains attention →
Generative AI in Action
Attention is a neural-network mechanism that assigns different importance weights to parts of an input when producing an output.
- How it is used
- Explains how Transformers select relevant language or image information for generation, recognition, translation, and summarization.
- How deep it goes
- Defines self-attention and its scaling behavior, illustrates weighting with examples, and stops short of mathematical derivation or implementation code.
- Examples
- Uses next-word prediction, machine translation, summarization, image patches, and a Vision Transformer architecture.
- What you can do after
- Evaluate attention-based architectures and choose context sizes with awareness of task benefits, computational cost, and latency.
Build AI-Enhanced Web Apps
A neural-network mechanism that weights relationships between different parts of an input when computing representations or predictions.
- How it is used
- Explains how models weigh relationships and context when processing sequences for language and other large-scale tasks.
- How deep it goes
- Defines self-attention and illustrates contextual dependencies, but stops short of implementation details or mathematical internals.
- Examples
- Uses a lion-and-antelope reference example and contrasts contextual attention with isolated word interpretation.
- What you can do after
- Evaluate when transformer-based models fit tasks requiring global context and relationships across sequences.
AI Agents in Action, Second Edition
An attention layer computes context-dependent interactions among input elements to form representations for prediction.
- How it is used
- Routes control dynamically among specialized modules in a cognitive agent architecture.
- How deep it goes
- Connects attention to context representations and strategy selection, but does not detail attention-layer computation.
- Examples
- Uses next-token generation and feedback from ambiguity or contradiction as attention-related illustrations.
- What you can do after
- Evaluate how an agent can redirect reasoning when evaluation detects ambiguity or contradiction.
Read how AI Agents in Action, Second Edition explains attention →
Investing for Programmers
An attention mechanism dynamically weights relationships between elements of an input when computing a model’s representations or outputs.
- How it is used
- Explains why LLMs support financial question answering, risk overviews, and other natural language processing tasks.
- How deep it goes
- Defines keyword-focused relationships and their role in transformers, but defers algorithmic and mathematical implementation details.
- Examples
- Contrasts examining every word relationship with focusing on essential keywords in sentences.
- What you can do after
- Evaluate LLM capabilities and limitations when applying them to financial research and risk analysis.
Causal AI
Attention is a neural-network mechanism that assigns different weights to parts of an input when computing a representation or producing an output.
- How it is used
- Explains how language models select contextually relevant parts of token sequences to produce coherent representations and outputs.
- How deep it goes
- Explains weighted focus and stacked layers from local relationships to global themes, but stops short of attention’s mathematical implementation.
- Examples
- Uses wildfire language to show contextual disambiguation and layered connections between words, phrases, and themes.
- What you can do after
- Evaluate/understand how attention builds contextual representations without treating them as verified causal explanations.
How Large Language Models Work
Attention is a transformer mechanism that computes context-dependent relationships by matching queries with keys and combining the corresponding values.
- How it is used
- Explains how transformer layers use context to predict the next token.
- How deep it goes
- Traces queries, keys, weights, and values, but defers the detailed mathematics.
- Examples
- Uses a Python dictionary analogy and a next-token prediction diagram.
- What you can do after
- Evaluate how context influences transformer token predictions without treating attention as exact lookup or guaranteed reasoning.
Read how How Large Language Models Work explains attention →
Introduction to Generative AI, Second Edition
Attention is a neural-network mechanism that computes the relative importance of input elements when producing a representation or output.
- How it is used
- Uses relative importance to provide context during sequence generation, especially for machine translation and language modeling.
- How deep it goes
- Explains weighting, context, long-term dependencies, and parallelization but stops short of implementation details or mathematics.
- Examples
- Uses machine translation, the sentence about a cat and mouse, and changing attention distributions for “it.”
- What you can do after
- Evaluate/understand how attention helps language models select relevant context and process sequences more efficiently.
Read how Introduction to Generative AI, Second Edition explains attention →
Knowledge Graphs and LLMs in Action
An attention mechanism is a technique that assigns learned importance weights to inputs so a model can prioritize the most relevant information.
- How it is used
- Improves graph message passing by weighting neighbors according to task-relevant relationships.
- How deep it goes
- Explains weighted aggregation, learnable scoring, multihead variants, transformer connections, and structural encoding with implementation listings.
- Examples
- Uses social friendships, molecular bonds and spatial arrangements, PyTorch attention modules, and degree-based structural encodings.
- What you can do after
- Build it from scratch using weighted neighbor aggregation, multihead modules, transformer-style projections, and structural features.
Read how Knowledge Graphs and LLMs in Action explains attention →
Coding with AI
An attention mechanism computes the relevance of elements in an input sequence so a model can emphasize information important to the current computation.
- How it is used
- Supports Transformer-based language processing by identifying relevant relationships across an entire input sequence.
- How deep it goes
- Explains contextual relationships, parallel sequence processing, and layered self-attention without detailing the underlying calculations.
- Examples
- Uses “The dog chased the cat” and “Jane helps Joe” to illustrate relationships and word positions.
- What you can do after
- Evaluate/understand how Transformers use word relationships and positions to interpret sequence context.
The Complete Obsolete Guide to Generative AI
An attention mechanism is a neural-network component that weights relationships among input elements when computing representations or outputs.
- How it is used
- Explains how transformer-based language models represent context and relationships while producing predictions or text.
- How deep it goes
- Defines attention conceptually as learned weighting for context, but stops short of implementation details or mathematical derivation.
- Examples
- Uses words or tokens, neural-network connections, and transformer-based LLMs as concrete reference points.
- What you can do after
- Understand how learned connection weights help an LLM model context and influence its outputs.
Read how The Complete Obsolete Guide to Generative AI explains attention →
The AI Pocket Book
The transformer computation that assigns attention scores and uses them to weigh contextual influence among tokens.
- How it is used
- Uses token relevance to contextualize and disambiguate each token from preceding tokens within the context window.
- How deep it goes
- Explains scores, projections, learned matrices, and embedding influence, but defers the mathematical details of the operations.
- Examples
- Uses “dog’s bark” and Figure 1.10 to show relevance and contextual influence.
- What you can do after
- Evaluate how attention uses token relevance to clarify meaning and alter contextualized embeddings.
Related concepts
- transformer architectureused by · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
- inner productuses · An inner product is a scalar-valued operation on two vectors that combines their corresponding components, summing their products and, for complex vectors, using conjugation.
- graph attention networkused by · A graph attention network is a graph neural network that uses attention weights to aggregate information from neighboring nodes, allowing their contributions to differ.
- cross-attentioncontrasts with · Cross-attention is an attention mechanism in which queries from one sequence or representation attend to keys and values from another.
- Pointer Networkused by · A Pointer Network is an attention-based sequence-to-sequence neural architecture that outputs positions or elements selected from its input sequence.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.