A span-corruption objective pretrains an encoder-decoder model by replacing contiguous token spans with unique sentinel markers, then training it to generate the missing spans. Unlike masking isolated tokens, it teaches sequence reconstruction across longer missing regions. section 5.1
Reconstructing missing spans
Given a cleaned web sequence such as “The engineer deployed <extra_id_0> after testing <extra_id_1>,” the model learns to generate the corresponding missing spans in order. This pretraining supports T5’s text-to-text treatment of translation, summarization, and question answering. section 5.1
The objective helps a generative model convert one sequence into another using learned internal knowledge. It also contributes to the strengths and limits of T5-style systems: broad generative behavior, but static knowledge and no inherent grounding in verifiable external evidence. section 5.1
