liveBook · Manning

Bidirectional language model

A bidirectional language model predicts or represents tokens using contextual information from both preceding and following tokens in a sequence.

How 2 Manning books explain it

Natural Language Processing in Action, Second Edition

A bidirectional language model represents or predicts text using contextual information from both preceding and following tokens.

How it is used
Supports noncausal prediction, especially masked-text recovery, OCR correction, and question answering.
How deep it goes
Explains bidirectional recurrence, contextual balance, and causal limitations, but stops short of a complete bidirectional training walkthrough.
Examples
Uses masked reports, OCR-corrupted text, question answering, BERT, and bidirectional RNNs as concrete examples.
What you can do after
Apply bidirectional models to recover masked text, correct OCR errors, and improve question-answering systems.

Read how Natural Language Processing in Action, Second Edition explains bidirectional language model →

Transfer Learning for Natural Language Processing

A language model that combines forward and backward language models to use context from both directions in a sequence.

How it is used
Builds word embeddings that use context from both directions in a sequence.
How deep it goes
Explains directional token prediction, the joint objective, parameter sharing, and task-specific combinations of LSTM states.
Examples
Uses “You can be” to illustrate backward prediction and relates LSTM layers to part-of-speech tagging and meaning disambiguation.
What you can do after
Understand how ELMo combines directional language models and uses their internal states to represent tokens.

Read how Transfer Learning for Natural Language Processing explains bidirectional language model →

Also covered in

What to know first

Related concepts

Questions about bidirectional language model

What is bidirectional language model?

A bidirectional language model predicts or represents tokens using contextual information from both preceding and following tokens in a sequence.

Which Manning books explain bidirectional language model?

Natural Language Processing in Action, Second Edition and Transfer Learning for Natural Language Processing.

What should I know before learning bidirectional language model?

Manning books that teach bidirectional language model usually introduce convolutional neural network first.

What is the difference between bidirectional language model and autoregressive model?

A bidirectional language model predicts or represents tokens using contextual information from both preceding and following tokens in a sequence. An autoregressive model represents or predicts each value in a sequence based on preceding values in that sequence.

What kinds of bidirectional language model are there?

Kinds of bidirectional language model include BERT.

What is bidirectional language model a kind of?

Bidirectional language model is a kind of language model. A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.