Attention
Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
Inside Deep Learning: Math, Algorithms, Models
A technique that assigns weights to inputs and combines them so a model can focus on relevant information.
- How it is used
- Uses learned weights to focus predictions on relevant items in bags, variable-length sequences, and translation inputs.
- How deep it goes
- Works from softmax-weighted averages through PyTorch modules, contextual scoring, padding masks, and a complete sequence-to-sequence implementation.
- Examples
- Uses largest-digit MNIST bags, variable-length bags, token-embedding classification, and English-to-French attention maps.
- What you can do after
- Build it from scratch in PyTorch, mask padded inputs, compare score functions, and inspect learned weights.
Read how Inside Deep Learning: Math, Algorithms, Models explains attention →
Learn Generative AI with PyTorch
A neural sequence-processing technique that assigns learned weights to relationships between elements, including self-relationships, to model relevance across a sequence.
- How it is used
- Uses attention as the sequence-relationship mechanism in translation, text generation, and music generation Transformers.
- How deep it goes
- Works through scaled dot-product calculations, tensor shapes, multihead splitting, causal masking, and a from-scratch implementation.
- Examples
- Illustrates attention with library retrieval, English-to-French translation, a four-token causal mask, and musical event sequences.
- What you can do after
- Builds attention from scratch in PyTorch for translation and causal text generation models.
Read how Learn Generative AI with PyTorch explains attention →
Sutskever's List
An attention mechanism computes weighted interactions between elements to focus a neural network computation on task-relevant information.
- How it is used
- Uses attention to bypass the fixed-context bottleneck in translation and connect distant positions in Transformers.
- How deep it goes
- Explains relevance scores, weights, and context vectors for Bahdanau attention; treats Transformer self-attention through direct connections and computational trade-offs rather than a derivation.
- Examples
- Illustrates attention with English–French word alignments, BLEU comparisons, and direct links between distant words.
- What you can do after
- Evaluate/understand when attention helps sequence models and why all-pairs self-attention becomes costly for long inputs.
Build AI Drug Discovery Pipelines
An attention mechanism computes input-dependent weights that determine how strongly elements of a sequence or representation influence one another.
- How it is used
- Uses attention to combine relevant molecular neighbors and distant protein residues for drug-target affinity and protein modeling.
- How deep it goes
- Explains query-key-value aggregation, self-attention and multi-head attention, then demonstrates graph layers and a small protein language model.
- Examples
- Illustrates attention with distant cysteines, masked amino acid prediction and heatmaps of protein-sequence attention heads.
- What you can do after
- Apply attention through GATConv and train a small protein language model, evaluating masked-token predictions on held-out sequences.
Read how Build AI Drug Discovery Pipelines explains attention →
Deep Learning with R, Third Edition
An attention layer computes context-dependent representations by allowing sequence elements to weight information from other elements.
- How it is used
- Provides selective information flow for translation, contextual text classification, and autoregressive generation.
- How deep it goes
- Explains dot-product scoring, learned projections, multi-head attention, masking, and encoder-decoder implementation.
- Examples
- Uses English-to-Spanish translation, contextual word meanings, and IMDb review classification to illustrate attention.
- What you can do after
- Apply it via Keras multi-head attention to build Transformer encoder and decoder blocks.
Read how Deep Learning with R, Third Edition explains attention →
Build an Advanced RAG Application (From Scratch)
An attention mechanism weights the importance of different elements in an input sequence when computing a representation for a particular element or output.
- How it is used
- Explains how Transformers relate sequence positions for contextual understanding, representation learning, and text generation.
- How deep it goes
- Defines self-attention and its encoder, decoder, masking, and context-window cost roles without detailing attention calculations.
- Examples
- Uses a fox-and-dog sentence and 8,192-to-32,768-token embedding context scenarios to illustrate attention behavior and cost.
- What you can do after
- Evaluate/understand why attention improves long-range context yet constrains embedding context length through quadratic computation.
Read how Build an Advanced RAG Application (From Scratch) explains attention →
Build a Text-to-Image Generator (from Scratch)
A neural-network mechanism that weights input elements according to their relevance when computing a representation or prediction.
- How it is used
- Weights relevant words in translation and image regions in classification or denoising.
- How deep it goes
- Works through scaled dot-product scores, masking, softmax, and weighted values in code; introduces multi-head splitting without displaying its full class definition.
- Examples
- Uses German-to-English translation, a PyTorch attention function, image-patch classification, and high-resolution flower-image denoising.
- What you can do after
- Build it from scratch in an encoder–decoder transformer and apply the mechanism to image patches or a denoising U-Net.
Read how Build a Text-to-Image Generator (from Scratch) explains attention →
Deep Learning with Python, Third Edition
A mechanism that assigns context-dependent weights to parts of a sequence so a model can emphasize some elements over others when processing input.
- How it is used
- Uses attention to pass contextual information across sequences in translation and Transformer models.
- How deep it goes
- Works from dot-product scores and weighted sums through multi-head attention, encoder and decoder blocks, and attention masks.
- Examples
- Uses English-to-Spanish translation, a sentence about a train station, and RoBERTa movie-review classification.
- What you can do after
- Builds dot-product attention from scratch and applies multi-head attention via Keras in Transformer encoder and decoder blocks.
Read how Deep Learning with Python, Third Edition explains attention →
Rearchitecting LLMs
Attention is a neural-network mechanism that computes input-dependent weights to prioritize information during processing.
- How it is used
- Guides Transformer inference optimization by locating contextualization, compute, and KV-cache memory costs.
- How deep it goes
- Explains QKV weighting and head structure, then details hook-based scoring and attention-module removal.
- Examples
- Uses the contextual meaning of “spring,” DistilGPT2 head dimensions, and Llama attention-layer scores.
- What you can do after
- Operate and configure attention optimization using PyTorch hooks and Hugging Face model structures, then evaluate memory, speed, and generation quality.
Transformers in Action
An attention mechanism dynamically assigns greater computational weight to input elements that are more relevant to the current representation or prediction.
- How it is used
- Motivates attention as the mechanism for contextual sequence representations in transformers and contrasts it with recurrent processing.
- How deep it goes
- Explains scaled dot-product calculations and multihead attention with simplified implementations, from projections through weighted outputs.
- Examples
- Illustrates contextual word relationships with English-to-French translation and compares gradients for an LSTM and transformer on a long sentence.
- What you can do after
- Builds a simplified scaled dot-product attention calculation and a multihead attention module from scratch.
Machines that Think
A neural-network mechanism that computes relevance weights over elements so a model can selectively use information pertinent to its current computation.
- How it is used
- Frames attention as selective access to context for sequence modeling and multimodal integration.
- How deep it goes
- Explains learned queries, keys, values and contextual combinations, then distinguishes self-attention from cross-attention.
- Examples
- Uses “The sun rises in the east” to illustrate token relevance and examines answer placement in long contexts.
- What you can do after
- Evaluate/understand how attention combines context and why available information may not reliably influence an answer.
Build a Large Language Model (From Scratch)
A neural-network mechanism that weights input tokens or values by relevance so the model can focus on important context.
- How it is used
- Deploys attention inside transformer language models to select relevant context while generating or representing sequences.
- How deep it goes
- Explains weighted sums, dot products, queries, keys, values, causal masks, dropout, and multi-head stacking, without formal derivations.
- Examples
- Demonstrates attention through simplified dot products, weighted context vectors, causal masks, dropout masks, and stacked attention modules.
- What you can do after
- Build it from scratch as simplified, causal, and multi-head attention modules for a GPT-like language model.
Read how Build a Large Language Model (From Scratch) explains attention →
Grokking AI Algorithms, Second Edition
Attention is a neural-network mechanism that weights relationships among input representations so relevant information contributes more strongly to a computation.
- How it is used
- Uses attention to build contextual token representations for language modeling and connect text prompts to image features.
- How deep it goes
- Works through single-head query, key and value projections, scaled scores, softmax weights, and context-vector calculations.
- Examples
- Traces relationships in a sentence about Alice, numerical token weights and context vectors, and a text prompt for a generated image.
- What you can do after
- Evaluate and understand how attention weights combine token information and how cross-attention links image features to text.
Read how Grokking AI Algorithms, Second Edition explains attention →
LLMs in Production
An attention mechanism dynamically weights relationships between elements of an input sequence according to their relevance to a current computation.
- How it is used
- Uses attention to connect sequence relevance with the transition from LSTMs to transformer language models.
- How deep it goes
- Explains query–key dot products, value weighting, attention patterns, and quadratic scaling; notes that encoder and decoder lengths complicate full implementations.
- Examples
- Illustrates attention with a NumPy query–key–value listing and an encoder diagram.
- What you can do after
- Evaluate attention patterns and context-window costs when designing language-model applications.
Natural Language Processing in Action, Second Edition
An attention mechanism is a sequence-processing operation that assigns different importance to input elements when computing a representation or output.
- How it is used
- Connects important patterns across long text sequences for transformer encoding and language generation.
- How deep it goes
- Explains sequence-wide matrix application, recurrence removal, fixed-length limits, and stackable encodings, but does not detail training.
- Examples
- Uses translation, long passages, token sequences such as “What is BERT?”, attention matrices, and stackable transformer diagrams.
- What you can do after
- Evaluate how transformers connect distant tokens while trading recurrence problems for fixed-length sequence constraints.
Read how Natural Language Processing in Action, Second Edition explains attention →
Build a DeepSeek Model (From Scratch)
Standard attention is an attention mechanism that computes interactions between query representations and key-value representations to combine relevant information.
- How it is used
- Explains token relationships and context vectors for transformer next-token prediction, motivating inference caching and latent attention.
- How deep it goes
- Walks through projections, scores, causal weights, context matrices, multiple heads, and repeated inference calculations with explicit tensor shapes.
- Examples
- Traces “The next day is” through a four-token attention head, then appends “bright” for a second inference step.
- What you can do after
- Evaluate and understand attention computations and identify repeated work during autoregressive inference.
Read how Build a DeepSeek Model (From Scratch) explains attention →
Deep Learning with PyTorch, Second Edition
A neural-network technique that helps a model focus on the most relevant parts of its input when computing representations.
- How it is used
- Uses attention to capture context and sequence relationships for text generation and image classification.
- How deep it goes
- Develops query, key, and value intuition into dot-product self-attention and introduces scaled dot-product attention.
- Examples
- Illustrates contextual character representations with car and care, and computes self-attention on two random inputs.
- What you can do after
- Build self-attention from linear transformations, dot products, softmax, and weighted values.
Read how Deep Learning with PyTorch, Second Edition explains attention →
Understanding AI
An attention mechanism dynamically assigns importance weights to input elements when computing a representation or output.
- How it is used
- Explains how language models select relevant context while predicting the next token.
- How deep it goes
- Explains weighted token relationships, long-range context, multi-head attention, and transformer-scale training without mathematical derivations.
- Examples
- Uses umbrella continuation and pronoun resolution in a performance-and-crowd sentence to show relevance weighting.
- What you can do after
- Evaluate/understand why a model can retain distant context yet still operates through learned pattern prediction.
Graph Neural Networks in Action
An attention mechanism assigns learned, data-dependent importance weights to inputs when computing a representation.
- How it is used
- Highlights influential graph neighbors when node importance may not be apparent from degree.
- How deep it goes
- Defines attention intuitively, distinguishes related forms, and points to GAT equations for a deeper account.
- Examples
- Uses a nonlinear-story analogy and a fake-review detection network to illustrate selective emphasis.
- What you can do after
- Apply it via PyTorch Geometric GAT or GATv2 layers and compare against a GCN baseline.
Read how Graph Neural Networks in Action explains attention →
Grokking AI Applications
An attention mechanism computes the relative relevance of input elements so a model can focus on the most informative parts of a sequence.
- How it is used
- Explains how transformer-based language models identify relevant tokens while processing natural-language sequences.
- How deep it goes
- Defines the relevance-focused mechanism and gives one example, but stops short of its neural implementation details.
- Examples
- Uses the question about Italy’s capital to show why relevant tokens receive greater focus.
- What you can do after
- Evaluate how attention helps a language model select informative sequence elements and recognize context-window constraints.
Optimization Algorithms: AI techniques for design, planning, and control problems
An attention mechanism computes weights over representations so a model can focus on selected input information when producing an output.
- How it is used
- Uses learned relevance weights to focus sequence models or graph models on the input information needed for each output.
- How deep it goes
- Defines selective focus, scores, and applications, but stops short of deriving the attention computation or training procedure.
- Examples
- Illustrates attention with long-sequence translation, graph neighbors, convex hull selection, and the output sequence [1, 4, 2].
- What you can do after
- Evaluate how attention prioritizes sequence segments or graph neighbors and understand its use in variable-sized output selection.
Timeless Algorithms, The Foundational Ideas
An attention mechanism weights relationships among elements to form context-dependent representations.
- How it is used
- Explains how attention constructs context through direct relationships instead of relying entirely on recurrent state propagation.
- How deep it goes
- Explains weighted access to encoder states and the architectural shift to self-attention, without detailing the underlying calculations.
- Examples
- Illustrates attention through translation alignment, cross-passage document relationships, and links between program calls and definitions.
- What you can do after
- Evaluate how direct relational access differs from recurrent context propagation and why efficient attention variants may reduce costs.
Read how Timeless Algorithms, The Foundational Ideas explains attention →
Fight Fraud with Machine Learning
An attention layer is a neural-network layer that computes attention weights over elements or time steps of its input.
- How it is used
- Uses learned weights to emphasize relevant sequence elements, image regions, text, or graph neighbors during prediction.
- How deep it goes
- Explains weighted context vectors, self-attention coefficients, multi-head attention, and graph aggregation without detailing training equations.
- Examples
- Uses sentence completion, dog-versus-cat pixel classification, phishing detection, and Bitcoin transaction graphs as examples.
- What you can do after
- Evaluate/understand how attention selects relevant sequence positions and supports transformer and graph-based fraud models.
Read how Fight Fraud with Machine Learning explains attention →
Domain-Specific Small Language Models
A sequence-modeling technique that weights relevant parts of an input more heavily than less relevant parts.
- How it is used
- Weights sequence information and serves as a focus for Transformer training, inference, and serving performance decisions.
- How deep it goes
- Defines attention and self-attention, then examines KV-cache behavior and GPU data movement without developing a formal derivation.
- Examples
- Compares cached and uncached GPT-Neo generation on a T4 GPU and lists the attention operation's matrix and memory steps.
- What you can do after
- Diagnose/debug with it by checking KV-cache use and assessing whether attention-related inference work is constrained by compute or memory movement.
Read how Domain-Specific Small Language Models explains attention →
Applied Reinforcement Learning
An attention mechanism computes input-dependent weights to combine representations according to their relevance to one another.
- How it is used
- Coordinates cooling decisions by sharing relevant information among interacting zones.
- How deep it goes
- Defines attention as weighted combinations and explains its role in zone representations and adaptive feature extraction.
- Examples
- Uses data center cooling zones and a manager checking neighboring temperatures and workloads.
- What you can do after
- Understand how attention can help a policy coordinate decisions across interacting zones.
Read how Applied Reinforcement Learning explains attention →
Building Specialized AI Systems
A neural-network component that assigns different weights to input elements when computing a representation or output.
- How it is used
- Builds context-aware token representations for generation, representation learning, and sequence-to-sequence processing.
- How deep it goes
- Explains queries, keys, values, softmax weights, and weighted outputs, while placing attention within Transformer blocks.
- Examples
- Demonstrates score calculation, softmax normalization, weighted value aggregation, and the recurrent encoder bottleneck.
- What you can do after
- Understand how Transformers combine token interactions into context-aware representations for later model specialization.
Read how Building Specialized AI Systems explains attention →
Machine Learning Algorithms in Depth
Attention is a technique that computes context-dependent weighted combinations of input representations according to their relevance to a query.
- How it is used
- Uses adaptive weights to give sequence outputs direct, context-dependent access to input representations.
- How deep it goes
- Derives query–key–value weighting and a matrix form, then connects them to a transformer classifier implemented with Keras/TensorFlow layers.
- Examples
- Illustrates attention with a neural-machine-translation heatmap and a Keras/TensorFlow transformer text classifier.
- What you can do after
- Apply it via Keras/TensorFlow in a transformer text classifier and interpret query–key weights as weighted access to values.
Read how Machine Learning Algorithms in Depth explains attention →
Generative AI in Action
Attention is a neural-network mechanism that assigns different importance weights to parts of an input when producing an output.
- How it is used
- Explains how Transformers select relevant language or image information for generation, recognition, translation, and summarization.
- How deep it goes
- Defines self-attention and its scaling behavior, illustrates weighting with examples, and stops short of mathematical derivation or implementation code.
- Examples
- Uses next-word prediction, machine translation, summarization, image patches, and a Vision Transformer architecture.
- What you can do after
- Evaluate attention-based architectures and choose context sizes with awareness of task benefits, computational cost, and latency.
Build AI-Enhanced Web Apps
A neural-network mechanism that weights relationships between different parts of an input when computing representations or predictions.
- How it is used
- Explains how models weigh relationships and context when processing sequences for language and other large-scale tasks.
- How deep it goes
- Defines self-attention and illustrates contextual dependencies, but stops short of implementation details or mathematical internals.
- Examples
- Uses a lion-and-antelope reference example and contrasts contextual attention with isolated word interpretation.
- What you can do after
- Evaluate when transformer-based models fit tasks requiring global context and relationships across sequences.
AI Agents in Action, Second Edition
An attention layer computes context-dependent interactions among input elements to form representations for prediction.
- How it is used
- Routes control dynamically among specialized modules in a cognitive agent architecture.
- How deep it goes
- Connects attention to context representations and strategy selection, but does not detail attention-layer computation.
- Examples
- Uses next-token generation and feedback from ambiguity or contradiction as attention-related illustrations.
- What you can do after
- Evaluate how an agent can redirect reasoning when evaluation detects ambiguity or contradiction.
Read how AI Agents in Action, Second Edition explains attention →
Investing for Programmers
An attention mechanism dynamically weights relationships between elements of an input when computing a model’s representations or outputs.
- How it is used
- Explains why LLMs support financial question answering, risk overviews, and other natural language processing tasks.
- How deep it goes
- Defines keyword-focused relationships and their role in transformers, but defers algorithmic and mathematical implementation details.
- Examples
- Contrasts examining every word relationship with focusing on essential keywords in sentences.
- What you can do after
- Evaluate LLM capabilities and limitations when applying them to financial research and risk analysis.
Causal AI
Attention is a neural-network mechanism that assigns different weights to parts of an input when computing a representation or producing an output.
- How it is used
- Explains how language models select contextually relevant parts of token sequences to produce coherent representations and outputs.
- How deep it goes
- Explains weighted focus and stacked layers from local relationships to global themes, but stops short of attention’s mathematical implementation.
- Examples
- Uses wildfire language to show contextual disambiguation and layered connections between words, phrases, and themes.
- What you can do after
- Evaluate/understand how attention builds contextual representations without treating them as verified causal explanations.
CUDA for LLMs
An attention mechanism computes weighted combinations of representations according to their relevance to a query.
- How it is used
- Computes token relationships in transformer workloads through parallel dot products.
- How deep it goes
- Defines QKᵀ scores, softmax weights, and weighted values, then traces intermediate matrices’ HBM traffic.
- Examples
- Uses 1,000-token and 8,192-token sequences to illustrate quadratic work and score-matrix memory growth.
- What you can do after
- Evaluate attention’s memory bottleneck by tracing score and weight matrices between HBM and computation.
How Large Language Models Work
Attention is a transformer mechanism that computes context-dependent relationships by matching queries with keys and combining the corresponding values.
- How it is used
- Explains how transformer layers use context to predict the next token.
- How deep it goes
- Traces queries, keys, weights, and values, but defers the detailed mathematics.
- Examples
- Uses a Python dictionary analogy and a next-token prediction diagram.
- What you can do after
- Evaluate how context influences transformer token predictions without treating attention as exact lookup or guaranteed reasoning.
Read how How Large Language Models Work explains attention →
Introduction to Generative AI, Second Edition
Attention is a neural-network mechanism that computes the relative importance of input elements when producing a representation or output.
- How it is used
- Uses relative importance to provide context during sequence generation, especially for machine translation and language modeling.
- How deep it goes
- Explains weighting, context, long-term dependencies, and parallelization but stops short of implementation details or mathematics.
- Examples
- Uses machine translation, the sentence about a cat and mouse, and changing attention distributions for “it.”
- What you can do after
- Evaluate/understand how attention helps language models select relevant context and process sequences more efficiently.
Read how Introduction to Generative AI, Second Edition explains attention →
Transfer Learning for Natural Language Processing
Attention is a neural-network technique that assigns focus to selected parts of an input when computing an output.
- How it is used
- Uses attention to improve machine-translation sequence-to-sequence models by focusing on input parts relevant to each output.
- How deep it goes
- Explains attention’s role and historical impact, but stops short of implementation or formal mechanics.
- Examples
- Uses translating a source sentence into a target sentence to illustrate attention’s focus on relevant input parts.
- What you can do after
- Understand how attention helps translation models focus on relevant input parts and how transformers extend the approach.
Read how Transfer Learning for Natural Language Processing explains attention →
Knowledge Graphs and LLMs in Action
An attention mechanism is a technique that assigns learned importance weights to inputs so a model can prioritize the most relevant information.
- How it is used
- Improves graph message passing by weighting neighbors according to task-relevant relationships.
- How deep it goes
- Explains weighted aggregation, learnable scoring, multihead variants, transformer connections, and structural encoding with implementation listings.
- Examples
- Uses social friendships, molecular bonds and spatial arrangements, PyTorch attention modules, and degree-based structural encodings.
- What you can do after
- Build it from scratch using weighted neighbor aggregation, multihead modules, transformer-style projections, and structural features.
Read how Knowledge Graphs and LLMs in Action explains attention →
Coding with AI
An attention mechanism computes the relevance of elements in an input sequence so a model can emphasize information important to the current computation.
- How it is used
- Supports Transformer-based language processing by identifying relevant relationships across an entire input sequence.
- How deep it goes
- Explains contextual relationships, parallel sequence processing, and layered self-attention without detailing the underlying calculations.
- Examples
- Uses “The dog chased the cat” and “Jane helps Joe” to illustrate relationships and word positions.
- What you can do after
- Evaluate/understand how Transformers use word relationships and positions to interpret sequence context.
The Complete Obsolete Guide to Generative AI
An attention mechanism is a neural-network component that weights relationships among input elements when computing representations or outputs.
- How it is used
- Explains how transformer-based language models represent context and relationships while producing predictions or text.
- How deep it goes
- Defines attention conceptually as learned weighting for context, but stops short of implementation details or mathematical derivation.
- Examples
- Uses words or tokens, neural-network connections, and transformer-based LLMs as concrete reference points.
- What you can do after
- Understand how learned connection weights help an LLM model context and influence its outputs.
Read how The Complete Obsolete Guide to Generative AI explains attention →
The AI Pocket Book
The transformer computation that assigns attention scores and uses them to weigh contextual influence among tokens.
- How it is used
- Uses token relevance to contextualize and disambiguate each token from preceding tokens within the context window.
- How deep it goes
- Explains scores, projections, learned matrices, and embedding influence, but defers the mathematical details of the operations.
- Examples
- Uses “dog’s bark” and Figure 1.10 to show relevance and contextual influence.
- What you can do after
- Evaluate how attention uses token relevance to clarify meaning and alter contextualized embeddings.
Also covered in
- Hugging Face in Action
- Crack Any Codebase with AIIn early access (MEAP)
- Machine Learning System Design
- Time Series Forecasting Using Foundation Models
- Local Image GenerationIn early access (MEAP)
- Data Without Labels
- AI-Powered Developer: Build software with ChatGPT and Copilot
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- recurrent neural networkA recurrent neural network is a neural network that processes ordered inputs while carrying and updating an internal state across successive steps.
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
Related concepts
- transformer architectureused by · A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- softmaxuses · Softmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.
- inner productuses · An inner product is a scalar-valued operation on two vectors that generalizes the dot product, which for real coordinate vectors is the sum of products of corresponding components.
- graph attention networkused by · A graph attention network is a graph neural network that uses attention weights to aggregate information from neighboring nodes, allowing their contributions to differ.
- cross-attentioncontrasts with · Cross-attention is an attention mechanism in which queries from one sequence or representation attend to keys and values from another.
- self-attentionincludes · Self-attention is an attention mechanism that relates positions within a sequence, allowing each position to incorporate information from other positions in that sequence.
Questions about attention
What is attention?
Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
Which Manning books explain attention?
Inside Deep Learning: Math, Algorithms, Models, Learn Generative AI with PyTorch, Sutskever's List, Build AI Drug Discovery Pipelines, Deep Learning with R, Third Edition, Build an Advanced RAG Application (From Scratch), Build a Text-to-Image Generator (from Scratch), Deep Learning with Python, Third Edition and 32 more.
What should I know before learning attention?
Manning books that teach attention usually introduce language model, transformer architecture, recurrent neural network and token first.
What is the difference between attention and cross-attention?
Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation. Cross-attention is an attention mechanism in which queries from one sequence or representation attend to keys and values from another.
What kinds of attention are there?
Kinds of attention include self-attention.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.







































