Causal masking prevents each sequence position from attending to later positions. In a four-token sequence, earlier positions can use only the tokens up to and including themselves. Adding the mask to attention compatibility logits assigns blocked positions a value of −∞; softmax then gives them zero weight. section 12.2.3
The masked scores are the original compatibility scores plus the mask: allowed relationships remain available, while blocked future positions receive −∞. section 12.2.3
During generation, a predicted token is appended and becomes available to the next prediction. During training, causal masking preserves the next-token objective while allowing positions to be calculated in parallel; generation itself remains sequential. Masking is one part of attention, not a complete explanation of a transformer’s behavior. section 12.2.3 section 12.2 section 12.3.2









