A bidirectional language model uses context from both earlier and later tokens. Unlike a causal model, it can infer a missing or damaged word because the surrounding text is available in both directions. section 10.2.2 section 8.1.3
Its practical boundary is straightforward:
- Treats text as noncausal when predicting words that occur before or within already known context. section 8.1.3
- Fits masked-language tasks, such as recovering illegible report text or correcting optical character recognition errors. section 8.1.3
- Cannot generate ordinary prose strictly left to right, because future-token information would not be available during normal causal generation. section 10.2.2
Recovering corrupted text
Given a report with an intentionally masked word, or a sentence damaged by OCR, a bidirectional model can use both the preceding and following words to infer the missing content. This broader context can improve question-answering accuracy by roughly 5%–10% compared with a default forward-direction model. section 8.1.3
