Salient span masking hides information-bearing spans, such as named entities or dates, so retrieved documents directly help predict the masked content. Masking function words lets grammar solve the task without rewarding retrieval. section 3.2
Why the masked span matters
For a sentence whose answer is “Mexico,” retrieval can supply a document describing a monarch’s migration to Mexico. REALM scored 38.2 Exact Match with salient spans, versus 35.3 for random spans and 32.3 for random masks on Natural Questions. section 3.2
The strategy creates a useful training signal:
- Cross-entropy loss measures whether retrieved context improves masked-token prediction. section 3.2
- That signal updates reader and dual-encoder retriever weights toward documents containing task-relevant information. section 3.2
