An attention mask is a tensor or pattern that controls which sequence positions a model may use. Padding masks hide filler tokens; causal masks hide future tokens during prediction. section 2.2.2 section 8.2.2
Apply the mask at the stage where irrelevant positions would otherwise affect attention:
- Pad variable-length sequences to a common batch length, then mark real tokens as active and padding positions as inactive. section 8.2.2
- Build a source mask such as (src != pad), adding dimensions so attention can broadcast it across the sequence. section 2.2.2
- Build a decoder target mask that combines padding protection with a causal pattern, preventing each position from seeing subsequent tokens. section 2.2.2 section 2.3.2
- Pass the mask into attention while generating; the decoder then predicts one token at a time without incorporating padding or disallowed future positions. section 12.3.2
Two padded captions
For ['[CLS]', 'two', 'dogs', 'run', '[SEP]', '[PAD]', '[PAD]', '[PAD]'], the mask is [1,1,1,1,1,0,0,0]. A caption without padding receives all ones, so every position remains visible. section 8.2.2
attention_mask=text_tokens.not_equal(1)[:, None, None, :] #1
attention_state=torch.zeros(size=(24,4,256,2048), #2
device=device)
image_tokens=torch.full((1, 256 + 1), #3
2 ** 14 - 1,dtype=torch.long,device=device)
Inspect the compact PyTorch construction that compares text tokens with the padding index and reshapes the result for decoder attention. section 12.3.2
