liveBook · Manning

Span-corruption objective

As explained in Retrieval Augmented Generation, The Foundational Ideas

A span-corruption objective trains a model to reconstruct contiguous spans of tokens replaced by sentinel markers.

A span-corruption objective pretrains an encoder-decoder model by replacing contiguous token spans with unique sentinel markers, then training it to generate the missing spans. Unlike masking isolated tokens, it teaches sequence reconstruction across longer missing regions. section 5.1

Reconstructing missing spans

Given a cleaned web sequence such as “The engineer deployed <extra_id_0> after testing <extra_id_1>,” the model learns to generate the corresponding missing spans in order. This pretraining supports T5’s text-to-text treatment of translation, summarization, and question answering. section 5.1

The objective helps a generative model convert one sequence into another using learned internal knowledge. It also contributes to the strengths and limits of T5-style systems: broad generative behavior, but static knowledge and no inherent grounding in verifiable external evidence. section 5.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains span replacement and reconstruction, but stops short of training-loss or optimization details.
Examples
Uses C4 pretraining and reconstructs contiguous missing spans marked by unique sentinel tokens.
What you can do after
Understand how T5 learns sequence-to-sequence behavior from corrupted text before downstream task training.
How it connects
Connects span corruption with T5, encoder-decoder models, text-to-text learning, and retrieval-augmented generation.

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.