liveBook · Manning

Attention

Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.

How 40 Manning books explain it

Inside Deep Learning: Math, Algorithms, Models

A technique that assigns weights to inputs and combines them so a model can focus on relevant information.

How it is used
Uses learned weights to focus predictions on relevant items in bags, variable-length sequences, and translation inputs.
How deep it goes
Works from softmax-weighted averages through PyTorch modules, contextual scoring, padding masks, and a complete sequence-to-sequence implementation.
Examples
Uses largest-digit MNIST bags, variable-length bags, token-embedding classification, and English-to-French attention maps.
What you can do after
Build it from scratch in PyTorch, mask padded inputs, compare score functions, and inspect learned weights.

Read how Inside Deep Learning: Math, Algorithms, Models explains attention →

Learn Generative AI with PyTorch

A neural sequence-processing technique that assigns learned weights to relationships between elements, including self-relationships, to model relevance across a sequence.

How it is used
Uses attention as the sequence-relationship mechanism in translation, text generation, and music generation Transformers.
How deep it goes
Works through scaled dot-product calculations, tensor shapes, multihead splitting, causal masking, and a from-scratch implementation.
Examples
Illustrates attention with library retrieval, English-to-French translation, a four-token causal mask, and musical event sequences.
What you can do after
Builds attention from scratch in PyTorch for translation and causal text generation models.

Read how Learn Generative AI with PyTorch explains attention →

Sutskever's List

An attention mechanism computes weighted interactions between elements to focus a neural network computation on task-relevant information.

How it is used
Uses attention to bypass the fixed-context bottleneck in translation and connect distant positions in Transformers.
How deep it goes
Explains relevance scores, weights, and context vectors for Bahdanau attention; treats Transformer self-attention through direct connections and computational trade-offs rather than a derivation.
Examples
Illustrates attention with English–French word alignments, BLEU comparisons, and direct links between distant words.
What you can do after
Evaluate/understand when attention helps sequence models and why all-pairs self-attention becomes costly for long inputs.

Read how Sutskever's List explains attention →

Build AI Drug Discovery Pipelines

An attention mechanism computes input-dependent weights that determine how strongly elements of a sequence or representation influence one another.

How it is used
Uses attention to combine relevant molecular neighbors and distant protein residues for drug-target affinity and protein modeling.
How deep it goes
Explains query-key-value aggregation, self-attention and multi-head attention, then demonstrates graph layers and a small protein language model.
Examples
Illustrates attention with distant cysteines, masked amino acid prediction and heatmaps of protein-sequence attention heads.
What you can do after
Apply attention through GATConv and train a small protein language model, evaluating masked-token predictions on held-out sequences.

Read how Build AI Drug Discovery Pipelines explains attention →

Deep Learning with R, Third Edition

An attention layer computes context-dependent representations by allowing sequence elements to weight information from other elements.

How it is used
Provides selective information flow for translation, contextual text classification, and autoregressive generation.
How deep it goes
Explains dot-product scoring, learned projections, multi-head attention, masking, and encoder-decoder implementation.
Examples
Uses English-to-Spanish translation, contextual word meanings, and IMDb review classification to illustrate attention.
What you can do after
Apply it via Keras multi-head attention to build Transformer encoder and decoder blocks.

Read how Deep Learning with R, Third Edition explains attention →

Build an Advanced RAG Application (From Scratch)

An attention mechanism weights the importance of different elements in an input sequence when computing a representation for a particular element or output.

How it is used
Explains how Transformers relate sequence positions for contextual understanding, representation learning, and text generation.
How deep it goes
Defines self-attention and its encoder, decoder, masking, and context-window cost roles without detailing attention calculations.
Examples
Uses a fox-and-dog sentence and 8,192-to-32,768-token embedding context scenarios to illustrate attention behavior and cost.
What you can do after
Evaluate/understand why attention improves long-range context yet constrains embedding context length through quadratic computation.

Read how Build an Advanced RAG Application (From Scratch) explains attention →

Build a Text-to-Image Generator (from Scratch)

A neural-network mechanism that weights input elements according to their relevance when computing a representation or prediction.

How it is used
Weights relevant words in translation and image regions in classification or denoising.
How deep it goes
Works through scaled dot-product scores, masking, softmax, and weighted values in code; introduces multi-head splitting without displaying its full class definition.
Examples
Uses German-to-English translation, a PyTorch attention function, image-patch classification, and high-resolution flower-image denoising.
What you can do after
Build it from scratch in an encoder–decoder transformer and apply the mechanism to image patches or a denoising U-Net.

Read how Build a Text-to-Image Generator (from Scratch) explains attention →

Deep Learning with Python, Third Edition

A mechanism that assigns context-dependent weights to parts of a sequence so a model can emphasize some elements over others when processing input.

How it is used
Uses attention to pass contextual information across sequences in translation and Transformer models.
How deep it goes
Works from dot-product scores and weighted sums through multi-head attention, encoder and decoder blocks, and attention masks.
Examples
Uses English-to-Spanish translation, a sentence about a train station, and RoBERTa movie-review classification.
What you can do after
Builds dot-product attention from scratch and applies multi-head attention via Keras in Transformer encoder and decoder blocks.

Read how Deep Learning with Python, Third Edition explains attention →

Rearchitecting LLMs

Attention is a neural-network mechanism that computes input-dependent weights to prioritize information during processing.

How it is used
Guides Transformer inference optimization by locating contextualization, compute, and KV-cache memory costs.
How deep it goes
Explains QKV weighting and head structure, then details hook-based scoring and attention-module removal.
Examples
Uses the contextual meaning of “spring,” DistilGPT2 head dimensions, and Llama attention-layer scores.
What you can do after
Operate and configure attention optimization using PyTorch hooks and Hugging Face model structures, then evaluate memory, speed, and generation quality.

Read how Rearchitecting LLMs explains attention →

Transformers in Action

An attention mechanism dynamically assigns greater computational weight to input elements that are more relevant to the current representation or prediction.

How it is used
Motivates attention as the mechanism for contextual sequence representations in transformers and contrasts it with recurrent processing.
How deep it goes
Explains scaled dot-product calculations and multihead attention with simplified implementations, from projections through weighted outputs.
Examples
Illustrates contextual word relationships with English-to-French translation and compares gradients for an LSTM and transformer on a long sentence.
What you can do after
Builds a simplified scaled dot-product attention calculation and a multihead attention module from scratch.

Read how Transformers in Action explains attention →

Machines that Think

A neural-network mechanism that computes relevance weights over elements so a model can selectively use information pertinent to its current computation.

How it is used
Frames attention as selective access to context for sequence modeling and multimodal integration.
How deep it goes
Explains learned queries, keys, values and contextual combinations, then distinguishes self-attention from cross-attention.
Examples
Uses “The sun rises in the east” to illustrate token relevance and examines answer placement in long contexts.
What you can do after
Evaluate/understand how attention combines context and why available information may not reliably influence an answer.

Read how Machines that Think explains attention →

Build a Large Language Model (From Scratch)

A neural-network mechanism that weights input tokens or values by relevance so the model can focus on important context.

How it is used
Deploys attention inside transformer language models to select relevant context while generating or representing sequences.
How deep it goes
Explains weighted sums, dot products, queries, keys, values, causal masks, dropout, and multi-head stacking, without formal derivations.
Examples
Demonstrates attention through simplified dot products, weighted context vectors, causal masks, dropout masks, and stacked attention modules.
What you can do after
Build it from scratch as simplified, causal, and multi-head attention modules for a GPT-like language model.

Read how Build a Large Language Model (From Scratch) explains attention →

Grokking AI Algorithms, Second Edition

Attention is a neural-network mechanism that weights relationships among input representations so relevant information contributes more strongly to a computation.

How it is used
Uses attention to build contextual token representations for language modeling and connect text prompts to image features.
How deep it goes
Works through single-head query, key and value projections, scaled scores, softmax weights, and context-vector calculations.
Examples
Traces relationships in a sentence about Alice, numerical token weights and context vectors, and a text prompt for a generated image.
What you can do after
Evaluate and understand how attention weights combine token information and how cross-attention links image features to text.

Read how Grokking AI Algorithms, Second Edition explains attention →

LLMs in Production

An attention mechanism dynamically weights relationships between elements of an input sequence according to their relevance to a current computation.

How it is used
Uses attention to connect sequence relevance with the transition from LSTMs to transformer language models.
How deep it goes
Explains query–key dot products, value weighting, attention patterns, and quadratic scaling; notes that encoder and decoder lengths complicate full implementations.
Examples
Illustrates attention with a NumPy query–key–value listing and an encoder diagram.
What you can do after
Evaluate attention patterns and context-window costs when designing language-model applications.

Read how LLMs in Production explains attention →

Natural Language Processing in Action, Second Edition

An attention mechanism is a sequence-processing operation that assigns different importance to input elements when computing a representation or output.

How it is used
Connects important patterns across long text sequences for transformer encoding and language generation.
How deep it goes
Explains sequence-wide matrix application, recurrence removal, fixed-length limits, and stackable encodings, but does not detail training.
Examples
Uses translation, long passages, token sequences such as “What is BERT?”, attention matrices, and stackable transformer diagrams.
What you can do after
Evaluate how transformers connect distant tokens while trading recurrence problems for fixed-length sequence constraints.

Read how Natural Language Processing in Action, Second Edition explains attention →

Build a DeepSeek Model (From Scratch)

Standard attention is an attention mechanism that computes interactions between query representations and key-value representations to combine relevant information.

How it is used
Explains token relationships and context vectors for transformer next-token prediction, motivating inference caching and latent attention.
How deep it goes
Walks through projections, scores, causal weights, context matrices, multiple heads, and repeated inference calculations with explicit tensor shapes.
Examples
Traces “The next day is” through a four-token attention head, then appends “bright” for a second inference step.
What you can do after
Evaluate and understand attention computations and identify repeated work during autoregressive inference.

Read how Build a DeepSeek Model (From Scratch) explains attention →

Deep Learning with PyTorch, Second Edition

A neural-network technique that helps a model focus on the most relevant parts of its input when computing representations.

How it is used
Uses attention to capture context and sequence relationships for text generation and image classification.
How deep it goes
Develops query, key, and value intuition into dot-product self-attention and introduces scaled dot-product attention.
Examples
Illustrates contextual character representations with car and care, and computes self-attention on two random inputs.
What you can do after
Build self-attention from linear transformations, dot products, softmax, and weighted values.

Read how Deep Learning with PyTorch, Second Edition explains attention →

Understanding AI

An attention mechanism dynamically assigns importance weights to input elements when computing a representation or output.

How it is used
Explains how language models select relevant context while predicting the next token.
How deep it goes
Explains weighted token relationships, long-range context, multi-head attention, and transformer-scale training without mathematical derivations.
Examples
Uses umbrella continuation and pronoun resolution in a performance-and-crowd sentence to show relevance weighting.
What you can do after
Evaluate/understand why a model can retain distant context yet still operates through learned pattern prediction.

Read how Understanding AI explains attention →

Graph Neural Networks in Action

An attention mechanism assigns learned, data-dependent importance weights to inputs when computing a representation.

How it is used
Highlights influential graph neighbors when node importance may not be apparent from degree.
How deep it goes
Defines attention intuitively, distinguishes related forms, and points to GAT equations for a deeper account.
Examples
Uses a nonlinear-story analogy and a fake-review detection network to illustrate selective emphasis.
What you can do after
Apply it via PyTorch Geometric GAT or GATv2 layers and compare against a GCN baseline.

Read how Graph Neural Networks in Action explains attention →

Grokking AI Applications

An attention mechanism computes the relative relevance of input elements so a model can focus on the most informative parts of a sequence.

How it is used
Explains how transformer-based language models identify relevant tokens while processing natural-language sequences.
How deep it goes
Defines the relevance-focused mechanism and gives one example, but stops short of its neural implementation details.
Examples
Uses the question about Italy’s capital to show why relevant tokens receive greater focus.
What you can do after
Evaluate how attention helps a language model select informative sequence elements and recognize context-window constraints.

Read how Grokking AI Applications explains attention →

Optimization Algorithms: AI techniques for design, planning, and control problems

An attention mechanism computes weights over representations so a model can focus on selected input information when producing an output.

How it is used
Uses learned relevance weights to focus sequence models or graph models on the input information needed for each output.
How deep it goes
Defines selective focus, scores, and applications, but stops short of deriving the attention computation or training procedure.
Examples
Illustrates attention with long-sequence translation, graph neighbors, convex hull selection, and the output sequence [1, 4, 2].
What you can do after
Evaluate how attention prioritizes sequence segments or graph neighbors and understand its use in variable-sized output selection.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains attention →

Timeless Algorithms, The Foundational Ideas

An attention mechanism weights relationships among elements to form context-dependent representations.

How it is used
Explains how attention constructs context through direct relationships instead of relying entirely on recurrent state propagation.
How deep it goes
Explains weighted access to encoder states and the architectural shift to self-attention, without detailing the underlying calculations.
Examples
Illustrates attention through translation alignment, cross-passage document relationships, and links between program calls and definitions.
What you can do after
Evaluate how direct relational access differs from recurrent context propagation and why efficient attention variants may reduce costs.

Read how Timeless Algorithms, The Foundational Ideas explains attention →

Fight Fraud with Machine Learning

An attention layer is a neural-network layer that computes attention weights over elements or time steps of its input.

How it is used
Uses learned weights to emphasize relevant sequence elements, image regions, text, or graph neighbors during prediction.
How deep it goes
Explains weighted context vectors, self-attention coefficients, multi-head attention, and graph aggregation without detailing training equations.
Examples
Uses sentence completion, dog-versus-cat pixel classification, phishing detection, and Bitcoin transaction graphs as examples.
What you can do after
Evaluate/understand how attention selects relevant sequence positions and supports transformer and graph-based fraud models.

Read how Fight Fraud with Machine Learning explains attention →

Domain-Specific Small Language Models

A sequence-modeling technique that weights relevant parts of an input more heavily than less relevant parts.

How it is used
Weights sequence information and serves as a focus for Transformer training, inference, and serving performance decisions.
How deep it goes
Defines attention and self-attention, then examines KV-cache behavior and GPU data movement without developing a formal derivation.
Examples
Compares cached and uncached GPT-Neo generation on a T4 GPU and lists the attention operation's matrix and memory steps.
What you can do after
Diagnose/debug with it by checking KV-cache use and assessing whether attention-related inference work is constrained by compute or memory movement.

Read how Domain-Specific Small Language Models explains attention →

Applied Reinforcement Learning

An attention mechanism computes input-dependent weights to combine representations according to their relevance to one another.

How it is used
Coordinates cooling decisions by sharing relevant information among interacting zones.
How deep it goes
Defines attention as weighted combinations and explains its role in zone representations and adaptive feature extraction.
Examples
Uses data center cooling zones and a manager checking neighboring temperatures and workloads.
What you can do after
Understand how attention can help a policy coordinate decisions across interacting zones.

Read how Applied Reinforcement Learning explains attention →

Building Specialized AI Systems

A neural-network component that assigns different weights to input elements when computing a representation or output.

How it is used
Builds context-aware token representations for generation, representation learning, and sequence-to-sequence processing.
How deep it goes
Explains queries, keys, values, softmax weights, and weighted outputs, while placing attention within Transformer blocks.
Examples
Demonstrates score calculation, softmax normalization, weighted value aggregation, and the recurrent encoder bottleneck.
What you can do after
Understand how Transformers combine token interactions into context-aware representations for later model specialization.

Read how Building Specialized AI Systems explains attention →

Machine Learning Algorithms in Depth

Attention is a technique that computes context-dependent weighted combinations of input representations according to their relevance to a query.

How it is used
Uses adaptive weights to give sequence outputs direct, context-dependent access to input representations.
How deep it goes
Derives query–key–value weighting and a matrix form, then connects them to a transformer classifier implemented with Keras/TensorFlow layers.
Examples
Illustrates attention with a neural-machine-translation heatmap and a Keras/TensorFlow transformer text classifier.
What you can do after
Apply it via Keras/TensorFlow in a transformer text classifier and interpret query–key weights as weighted access to values.

Read how Machine Learning Algorithms in Depth explains attention →

Generative AI in Action

Attention is a neural-network mechanism that assigns different importance weights to parts of an input when producing an output.

How it is used
Explains how Transformers select relevant language or image information for generation, recognition, translation, and summarization.
How deep it goes
Defines self-attention and its scaling behavior, illustrates weighting with examples, and stops short of mathematical derivation or implementation code.
Examples
Uses next-word prediction, machine translation, summarization, image patches, and a Vision Transformer architecture.
What you can do after
Evaluate attention-based architectures and choose context sizes with awareness of task benefits, computational cost, and latency.

Read how Generative AI in Action explains attention →

Build AI-Enhanced Web Apps

A neural-network mechanism that weights relationships between different parts of an input when computing representations or predictions.

How it is used
Explains how models weigh relationships and context when processing sequences for language and other large-scale tasks.
How deep it goes
Defines self-attention and illustrates contextual dependencies, but stops short of implementation details or mathematical internals.
Examples
Uses a lion-and-antelope reference example and contrasts contextual attention with isolated word interpretation.
What you can do after
Evaluate when transformer-based models fit tasks requiring global context and relationships across sequences.

Read how Build AI-Enhanced Web Apps explains attention →

AI Agents in Action, Second Edition

An attention layer computes context-dependent interactions among input elements to form representations for prediction.

How it is used
Routes control dynamically among specialized modules in a cognitive agent architecture.
How deep it goes
Connects attention to context representations and strategy selection, but does not detail attention-layer computation.
Examples
Uses next-token generation and feedback from ambiguity or contradiction as attention-related illustrations.
What you can do after
Evaluate how an agent can redirect reasoning when evaluation detects ambiguity or contradiction.

Read how AI Agents in Action, Second Edition explains attention →

Investing for Programmers

An attention mechanism dynamically weights relationships between elements of an input when computing a model’s representations or outputs.

How it is used
Explains why LLMs support financial question answering, risk overviews, and other natural language processing tasks.
How deep it goes
Defines keyword-focused relationships and their role in transformers, but defers algorithmic and mathematical implementation details.
Examples
Contrasts examining every word relationship with focusing on essential keywords in sentences.
What you can do after
Evaluate LLM capabilities and limitations when applying them to financial research and risk analysis.

Read how Investing for Programmers explains attention →

Causal AI

Attention is a neural-network mechanism that assigns different weights to parts of an input when computing a representation or producing an output.

How it is used
Explains how language models select contextually relevant parts of token sequences to produce coherent representations and outputs.
How deep it goes
Explains weighted focus and stacked layers from local relationships to global themes, but stops short of attention’s mathematical implementation.
Examples
Uses wildfire language to show contextual disambiguation and layered connections between words, phrases, and themes.
What you can do after
Evaluate/understand how attention builds contextual representations without treating them as verified causal explanations.

Read how Causal AI explains attention →

CUDA for LLMs

An attention mechanism computes weighted combinations of representations according to their relevance to a query.

How it is used
Computes token relationships in transformer workloads through parallel dot products.
How deep it goes
Defines QKᵀ scores, softmax weights, and weighted values, then traces intermediate matrices’ HBM traffic.
Examples
Uses 1,000-token and 8,192-token sequences to illustrate quadratic work and score-matrix memory growth.
What you can do after
Evaluate attention’s memory bottleneck by tracing score and weight matrices between HBM and computation.

Read how CUDA for LLMs explains attention →

How Large Language Models Work

Attention is a transformer mechanism that computes context-dependent relationships by matching queries with keys and combining the corresponding values.

How it is used
Explains how transformer layers use context to predict the next token.
How deep it goes
Traces queries, keys, weights, and values, but defers the detailed mathematics.
Examples
Uses a Python dictionary analogy and a next-token prediction diagram.
What you can do after
Evaluate how context influences transformer token predictions without treating attention as exact lookup or guaranteed reasoning.

Read how How Large Language Models Work explains attention →

Introduction to Generative AI, Second Edition

Attention is a neural-network mechanism that computes the relative importance of input elements when producing a representation or output.

How it is used
Uses relative importance to provide context during sequence generation, especially for machine translation and language modeling.
How deep it goes
Explains weighting, context, long-term dependencies, and parallelization but stops short of implementation details or mathematics.
Examples
Uses machine translation, the sentence about a cat and mouse, and changing attention distributions for “it.”
What you can do after
Evaluate/understand how attention helps language models select relevant context and process sequences more efficiently.

Read how Introduction to Generative AI, Second Edition explains attention →

Transfer Learning for Natural Language Processing

Attention is a neural-network technique that assigns focus to selected parts of an input when computing an output.

How it is used
Uses attention to improve machine-translation sequence-to-sequence models by focusing on input parts relevant to each output.
How deep it goes
Explains attention’s role and historical impact, but stops short of implementation or formal mechanics.
Examples
Uses translating a source sentence into a target sentence to illustrate attention’s focus on relevant input parts.
What you can do after
Understand how attention helps translation models focus on relevant input parts and how transformers extend the approach.

Read how Transfer Learning for Natural Language Processing explains attention →

Knowledge Graphs and LLMs in Action

An attention mechanism is a technique that assigns learned importance weights to inputs so a model can prioritize the most relevant information.

How it is used
Improves graph message passing by weighting neighbors according to task-relevant relationships.
How deep it goes
Explains weighted aggregation, learnable scoring, multihead variants, transformer connections, and structural encoding with implementation listings.
Examples
Uses social friendships, molecular bonds and spatial arrangements, PyTorch attention modules, and degree-based structural encodings.
What you can do after
Build it from scratch using weighted neighbor aggregation, multihead modules, transformer-style projections, and structural features.

Read how Knowledge Graphs and LLMs in Action explains attention →

Coding with AI

An attention mechanism computes the relevance of elements in an input sequence so a model can emphasize information important to the current computation.

How it is used
Supports Transformer-based language processing by identifying relevant relationships across an entire input sequence.
How deep it goes
Explains contextual relationships, parallel sequence processing, and layered self-attention without detailing the underlying calculations.
Examples
Uses “The dog chased the cat” and “Jane helps Joe” to illustrate relationships and word positions.
What you can do after
Evaluate/understand how Transformers use word relationships and positions to interpret sequence context.

Read how Coding with AI explains attention →

The Complete Obsolete Guide to Generative AI

An attention mechanism is a neural-network component that weights relationships among input elements when computing representations or outputs.

How it is used
Explains how transformer-based language models represent context and relationships while producing predictions or text.
How deep it goes
Defines attention conceptually as learned weighting for context, but stops short of implementation details or mathematical derivation.
Examples
Uses words or tokens, neural-network connections, and transformer-based LLMs as concrete reference points.
What you can do after
Understand how learned connection weights help an LLM model context and influence its outputs.

Read how The Complete Obsolete Guide to Generative AI explains attention →

The AI Pocket Book

The transformer computation that assigns attention scores and uses them to weigh contextual influence among tokens.

How it is used
Uses token relevance to contextualize and disambiguate each token from preceding tokens within the context window.
How deep it goes
Explains scores, projections, learned matrices, and embedding influence, but defers the mathematical details of the operations.
Examples
Uses “dog’s bark” and Figure 1.10 to show relevance and contextual influence.
What you can do after
Evaluate how attention uses token relevance to clarify meaning and alter contextualized embeddings.

Read how The AI Pocket Book explains attention →

Also covered in

What to know first

Related concepts

Questions about attention

What is attention?

Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.

Which Manning books explain attention?

Inside Deep Learning: Math, Algorithms, Models, Learn Generative AI with PyTorch, Sutskever's List, Build AI Drug Discovery Pipelines, Deep Learning with R, Third Edition, Build an Advanced RAG Application (From Scratch), Build a Text-to-Image Generator (from Scratch), Deep Learning with Python, Third Edition and 32 more.

What should I know before learning attention?

Manning books that teach attention usually introduce language model, transformer architecture, recurrent neural network and token first.

What is the difference between attention and cross-attention?

Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation. Cross-attention is an attention mechanism in which queries from one sequence or representation attend to keys and values from another.

What kinds of attention are there?

Kinds of attention include self-attention.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.