The [MASK] token is a special placeholder that intentionally hides a known input token so an encoder can predict what belongs there. It turns a sentence into a small reconstruction puzzle: the surrounding tokens provide context, while the model predicts the missing token ID. section 9.3.1 section 9.2.3
Masked language model pretraining uses the token in a controlled replacement scheme:
- Select 15% of WordPiece tokens for masking. section 9.3.1
- Replace selected tokens with [MASK] 80% of the time. section 9.3.1
- Replace them with a random token 10% of the time and keep the original token 10% of the time. section 9.3.1
- Reserve the marker as a tokenizer special token, commonly written <mask>, rather than as an ordinary vocabulary word. section 9.2.3
Reconstructing a hidden word
Given “The model reads [MASK] text,” the encoder uses the visible context to predict the hidden token. The replacement process also limits dependence on seeing [MASK] during later fine-tuning, where the marker is absent. section 9.3.1
Figure 9.11 Combining token semantics, context semantics, and token position information
Inspect how token, segment, and positional embeddings are summed into each input representation. section 9.3.1
