Bidirectional language model
A bidirectional language model predicts or represents tokens using contextual information from both preceding and following tokens in a sequence.
Natural Language Processing in Action, Second Edition
A bidirectional language model represents or predicts text using contextual information from both preceding and following tokens.
- How it is used
- Supports noncausal prediction, especially masked-text recovery, OCR correction, and question answering.
- How deep it goes
- Explains bidirectional recurrence, contextual balance, and causal limitations, but stops short of a complete bidirectional training walkthrough.
- Examples
- Uses masked reports, OCR-corrupted text, question answering, BERT, and bidirectional RNNs as concrete examples.
- What you can do after
- Apply bidirectional models to recover masked text, correct OCR errors, and improve question-answering systems.
Transfer Learning for Natural Language Processing
A language model that combines forward and backward language models to use context from both directions in a sequence.
- How it is used
- Builds word embeddings that use context from both directions in a sequence.
- How deep it goes
- Explains directional token prediction, the joint objective, parameter sharing, and task-specific combinations of LSTM states.
- Examples
- Uses “You can be” to illustrate backward prediction and relates LSTM layers to part-of-speech tagging and meaning disambiguation.
- What you can do after
- Understand how ELMo combines directional language models and uses their internal states to represent tokens.
Read how Transfer Learning for Natural Language Processing explains bidirectional language model →
Related concepts
- BERTincludes · BERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
- autoregressive modelcontrasts with · An autoregressive model represents or predicts each value in a sequence based on preceding values in that sequence.
- word embeddingproduces · A representation of a word as a vector of real numbers, or a technique for learning such representations.
- ELMoused by · ELMo (Embeddings from Language Models) is a deep contextualized word-representation model that generates context-dependent embeddings from a bidirectional language model.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

