liveBook · Manning

Salient span masking

As explained in Retrieval Augmented Generation, The Foundational Ideas

Salient span masking is a masking strategy that hides information-bearing spans such as named entities or dates so retrieval affects prediction.

Salient span masking hides information-bearing spans, such as named entities or dates, so retrieved documents directly help predict the masked content. Masking function words lets grammar solve the task without rewarding retrieval. section 3.2

Why the masked span matters

For a sentence whose answer is “Mexico,” retrieval can supply a document describing a monarch’s migration to Mexico. REALM scored 38.2 Exact Match with salient spans, versus 35.3 for random spans and 32.3 for random masks on Natural Questions. section 3.2

The strategy creates a useful training signal:

  • Cross-entropy loss measures whether retrieved context improves masked-token prediction. section 3.2
  • That signal updates reader and dual-encoder retriever weights toward documents containing task-relevant information. section 3.2

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the masking rationale, learning signal, and ablation results, but does not detail span-selection implementation.
Examples
Uses named entities, dates, function words, and a monarch’s migration to Mexico, with Natural Questions Exact Match comparisons.
What you can do after
Evaluate masking strategies by whether retrieval contributes to prediction and by comparing downstream Exact Match results.
How it connects
Connects masked language modeling with retrieval, readers, dual encoders, cross-entropy loss, and end-to-end training.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.