Masked attention, also called causal attention, prevents a sequence position from attending to later tokens. This preserves causality: a prediction can use the context available so far, but not future tokens. section 2.2.1 section 3.2
In an encoder–decoder transformer, masked self-attention helps the decoder generate outputs in order; a separate attention sublayer lets it use the encoder’s output. In a decoder-only model, causal attention supports autoregressive generation: each generated token is fed back into the input to predict the next one. section 2.2.1 section 3.2
Predicting from preceding context
Given the input tokens A, B, and C, a causal mask lets the position at B attend to A and B, but not C. The position at C can use A, B, and C to predict the next token. This matches the next-token prediction objective used to train decoder-only models. section 3.2
A mask can be constructed by comparing query and key positions. For equal sequence lengths, the condition i >= j marks the current and earlier positions as allowed. To suppress future positions, add a large negative value to their attention weights before they are used. section 3.2
Listing 3.2 Masked multihead self-attention in decoder-only models
def attention_mask(nd, ns, *, dtype):
i = tf.range(nd)[:, None] #1
j = tf.range(ns)
m = i >= j - ns + nd
return tf.cast(m, dtype)
def mask_attn_weights(w):
_, nd, ns = shape_list(w) #2
b = attention_mask(nd, ns, dtype=w.dtype) #3
b = tf.reshape(b, [1, 1, nd, ns]) #4
w = w + b * -tf.cast(1e10, w.dtype) * (1 - b) #5
return w
The listing constructs and reshapes a causal mask, but its final expression multiplies b by (1 - b). Because b is binary, that product is always zero and the expression does not suppress future positions. The intended addition would omit the leading b: w + (1 - b) * -tf.cast(1e10, w.dtype). section 3.2
Causal masking differs from bidirectional self-attention, which encoder-only models use to build contextual embeddings. Padding masking is another masking use case; it serves a different purpose from preserving generation order. section 3.6 section 2.2.1









