Masked attention, also called causal attention, prevents a sequence position from attending to future tokens. A decoder can therefore predict using only context available at that position. In autoregressive generation, each new token is fed back into the input stream to help predict the next one. section 3.2
The constraint distinguishes decoder self-attention from unrestricted attention over a sequence. In an encoder–decoder transformer, the decoder also attends to the encoder’s output, so it can use input-text context while generating output in order. A decoder-only model instead relies on its prior outputs. section 2.2.1 section 3.2
Which positions are available?
When predicting from the second position, a later third position must be unavailable. Once generation reaches the third position, the earlier outputs can provide context. This is the restriction behind next-token prediction rather than a rule that removes earlier context. section 2.2.1 section 3.2
Listing 3.2 Masked multihead self-attention in decoder-only models
def attention_mask(nd, ns, *, dtype):
i = tf.range(nd)[:, None] #1
j = tf.range(ns)
m = i >= j - ns + nd
return tf.cast(m, dtype)
def mask_attn_weights(w):
_, nd, ns = shape_list(w) #2
b = attention_mask(nd, ns, dtype=w.dtype) #3
b = tf.reshape(b, [1, 1, nd, ns]) #4
w = w + b * -tf.cast(1e10, w.dtype) * (1 - b) #5
return w
The TensorFlow listing builds a position mask, reshapes it for attention weights, and attempts to suppress disallowed weights. Check the final update carefully: for a binary mask b, its factor b × (1 − b) is always zero, so the expression as printed does not suppress any weights. section 3.2
