liveBook · Manning

Attention_mask

Also called padding mask

An attention mask is a tensor or indicator structure specifying which sequence positions may participate in attention or other model computations, often excluding padding.

How 3 Manning books explain it

Build AI Drug Discovery Pipelines

An attention mask is a tensor or rule that marks which sequence positions should contribute to attention or aggregation computations.

How it is used
Uses valid-position markers to keep padding tokens out of pooled protein-sequence representations.
How deep it goes
Implements expansion, masking, summation, denominator clamping, and division for mean pooling.
Examples
Demonstrates masked average token embeddings for variable-length antibody sequences.
What you can do after
Apply it via PyTorch to compute pooled sequence representations without padding contamination.

Read how Build AI Drug Discovery Pipelines explains attention_mask →

Build a Text-to-Image Generator (from Scratch)

A tensor or pattern that hides selected sequence positions from a model during processing or prediction.

How it is used
Controls padding visibility and autoregressive information flow in encoder-decoder training, text tokenization, and image-token generation.
How deep it goes
Implements binary padding masks, broadcast dimensions, and causal decoder masking, but does not detail attention-score masking internals.
Examples
Uses padded captions, German-English batches, decoder targets, and 256-token image generation.
What you can do after
Implement it with PyTorch tensors to exclude padding and prevent future-token visibility during decoding.

Read how Build a Text-to-Image Generator (from Scratch) explains attention_mask →

Deep Learning with R, Third Edition

A padding mask is a tensor of indicators identifying padded positions that should be ignored during sequence computation or loss evaluation.

How it is used
Prevents Transformer attention from using padding tokens and tracks padded positions during language-model preprocessing.
How deep it goes
Defines Boolean masks, broadcasting shapes, and causal alternatives, but does not derive attention-score calculations.
Examples
Uses encoder source masks, decoder lower-triangular masks, and 2-by-512 preprocessing outputs containing token IDs and padding masks.
What you can do after
Operate/configure Keras multi-head attention with padding or causal masks and inspect preprocessing masks before training.

Read how Deep Learning with R, Third Edition explains attention_mask →

Also covered in

What to know first

Related concepts

Questions about attention_mask

What is attention_mask?

An attention mask is a tensor or indicator structure specifying which sequence positions may participate in attention or other model computations, often excluding padding.

Which Manning books explain attention_mask?

Build AI Drug Discovery Pipelines, Build a Text-to-Image Generator (from Scratch) and Deep Learning with R, Third Edition.

What should I know before learning attention_mask?

Manning books that teach attention_mask usually introduce foundation model, transformer architecture, adam and language model first.

What is the difference between attention_mask and padding token?

An attention mask is a tensor or indicator structure specifying which sequence positions may participate in attention or other model computations, often excluding padding. A padding token is a special placeholder added to a sequence to bring it to a required or common length without representing meaningful sequence content.

What is attention_mask also called?

Attention_mask is also called padding mask.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.