Attention_mask
An attention mask is a tensor or indicator structure specifying which sequence positions may participate in attention or other model computations, often excluding padding.
Build a Text-to-Image Generator (from Scratch)
A tensor or pattern that hides selected sequence positions from a model during processing or prediction.
- How it is used
- Controls padding visibility and autoregressive information flow in encoder-decoder training, text tokenization, and image-token generation.
- How deep it goes
- Implements binary padding masks, broadcast dimensions, and causal decoder masking, but does not detail attention-score masking internals.
- Examples
- Uses padded captions, German-English batches, decoder targets, and 256-token image generation.
- What you can do after
- Implement it with PyTorch tensors to exclude padding and prevent future-token visibility during decoding.
Read how Build a Text-to-Image Generator (from Scratch) explains attention_mask →
Deep Learning with R, Third Edition
A padding mask is a tensor of indicators identifying padded positions that should be ignored during sequence computation or loss evaluation.
- How it is used
- Prevents Transformer attention from using padding tokens and tracks padded positions during language-model preprocessing.
- How deep it goes
- Defines Boolean masks, broadcasting shapes, and causal alternatives, but does not derive attention-score calculations.
- Examples
- Uses encoder source masks, decoder lower-triangular masks, and 2-by-512 preprocessing outputs containing token IDs and padding masks.
- What you can do after
- Operate/configure Keras multi-head attention with padding or causal masks and inspect preprocessing masks before training.
Read how Deep Learning with R, Third Edition explains attention_mask →
Related concepts
- multi-head attentionconfigures · Multi-head attention is an attention mechanism that computes multiple attention operations in parallel using distinct learned projections and combines their outputs.
- next-token predictionconfigures · Next-token prediction is the task or training objective of estimating the token that follows a sequence, given its preceding tokens and any conditioning information.
- padding tokencontrasts with · A padding token is a special placeholder inserted into shorter sequences to give batched sequences a uniform length without representing meaningful content.
- decodeconfigures · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
- cross-attentionused by · Cross-attention is an attention mechanism in which queries from one sequence or representation attend to keys and values from another.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.