liveBook · Manning

Masked attention

As explained in Transformers in Action

Causal masking restricts a sequence position from attending to later positions so predictions depend only on preceding context.

Masked attention, also called causal attention, prevents a sequence position from attending to later tokens. This preserves causality: a prediction can use the context available so far, but not future tokens. section 2.2.1 section 3.2

In an encoder–decoder transformer, masked self-attention helps the decoder generate outputs in order; a separate attention sublayer lets it use the encoder’s output. In a decoder-only model, causal attention supports autoregressive generation: each generated token is fed back into the input to predict the next one. section 2.2.1 section 3.2

Predicting from preceding context

Given the input tokens A, B, and C, a causal mask lets the position at B attend to A and B, but not C. The position at C can use A, B, and C to predict the next token. This matches the next-token prediction objective used to train decoder-only models. section 3.2

A mask can be constructed by comparing query and key positions. For equal sequence lengths, the condition i >= j marks the current and earlier positions as allowed. To suppress future positions, add a large negative value to their attention weights before they are used. section 3.2

Listing 3.2 Masked multihead self-attention in decoder-only models
def attention_mask(nd, ns, *, dtype):

    i = tf.range(nd)[:, None]   #1
    j = tf.range(ns)
    m = i >= j - ns + nd
    return tf.cast(m, dtype)

def mask_attn_weights(w):

    _, nd, ns = shape_list(w)   #2

    b = attention_mask(nd, ns, dtype=w.dtype)   #3

    b = tf.reshape(b, [1, 1, nd, ns])   #4

    w = w + b * -tf.cast(1e10, w.dtype) * (1 - b)  #5
    return w

The listing constructs and reshapes a causal mask, but its final expression multiplies b by (1 - b). Because b is binary, that product is always zero and the expression does not suppress future positions. The intended addition would omit the leading b: w + (1 - b) * -tf.cast(1e10, w.dtype). section 3.2

Causal masking differs from bidirectional self-attention, which encoder-only models use to build contextual embeddings. Padding masking is another masking use case; it serves a different purpose from preserving generation order. section 3.6 section 2.2.1

How Transformers in Action covers it

How deep it goes
Defines the causal restriction and illustrates mask construction and application in TensorFlow, although the supplied application expression does not suppress future positions.
Examples
Illustrates a causal position mask with a TensorFlow masked multihead self-attention listing.
What you can do after
Evaluate and understand how causal masking restricts context during token generation.
How it connects
Links masked attention to autoregressive generation, next-token prediction, encoder–decoder attention, and bidirectional self-attention.

Also explained in

Compare how 10 books explain masked attention →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.