Autoencoding trains a model to reconstruct corrupted inputs using surrounding context. It is one of three language-modeling objectives, alongside sequence-to-sequence transduction and autoregression; each requires broad linguistic knowledge but favors different tasks. section 5.1.2
The surrounding-context requirement matters because words can depend on both preceding and following positions. This contrasts with autoregression, which predicts the next token from previous tokens and therefore receives a one-sided learning signal. section 5.1.2
Context determines the missing word
A language model can use the words around a missing position to infer a plausible reconstruction, rather than relying only on the tokens that came before it. Figure 5.3 visualizes the broader context-prediction objective. section 5.1.2
