BERT

As explained in Natural Language Processing in Action, Second Edition

BERT is a transformer language model pretrained with bidirectional contextual objectives for representing and understanding text.

BERT is a bidirectional transformer encoder that learns contextual text representations by restoring masked tokens. Unlike causal models, it can use words on both sides of an interior gap, making it useful for understanding text rather than generating it left to right. section 9.3 section 10.3.11

Build each input representation from several coordinated signals:

• Tokenize with WordPiece and begin every sequence with [CLS]. section 9.3.1
• Separate one or two sentences with [SEP] and assign sentence-A or sentence-B segment embeddings. section 9.3.1
• Add token, segment, and positional embeddings together before passing the sequence through stacked encoder layers. section 9.3.1 section 9.3.3
• Pretrain with masked language modeling and next sentence prediction, then fine-tune all parameters end to end for a target task. section 9.3.1 section 9.3.2

Masked-token understanding

For a sentence containing an unknown interior word, mask selected WordPiece tokens and train BERT to recover their original IDs using surrounding context. A question-answering system can then fine-tune BERT to select an answer span from a passage, as in SQuAD. section 9.3 section 9.3.2

Figure 9.11 Combining token semantics, context semantics, and token position information

Inspect how token semantics, sentence context, and token positions combine into BERT's input representation. section 9.3.1

How Natural Language Processing in Action, Second Edition covers it

How deep it goes

Explains masking, tokenization, embeddings, pretraining, fine-tuning, and model sizes, while stopping short of exhaustive hardware guidance.

Examples

Uses masked words, sentence pairs, GLUE, SQuAD, PyTorch architecture summaries, and Haystack retrieval-generation pipelines.

What it assumes

Assumes familiarity with transformer layers and deep learning while explaining bidirectionality, masking, tokenization, and fine-tuning incrementally.

What you can do after

Apply it via fine-tuning to classify sentence pairs, select answer spans, or encode text for semantic retrieval.

How it connects

Connects transformers, WordPiece embeddings, masked language modeling, next sentence prediction, semantic search, GPT-2, and question answering.

liveBookmanning.com

Read Natural Language Processing in Action, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  LLMs in ProductionA transformer-based language representation model that uses bidirectional context to learn text embeddings.
• 

  liveBookmanning.com
  AI-Powered SearchBERT is a transformer-based language model pretrained to produce contextual representations of text using bidirectional attention.
• 

  liveBookmanning.com
  Machine Learning System DesignBERT is a pretrained bidirectional transformer language model that produces contextual representations of text.
• 

  liveBookmanning.com
  Domain-Specific Small Language ModelsA transformer-based language model that uses the encoder stack for understanding tasks such as classification.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthBERT is a pretrained bidirectional transformer language model designed to learn contextual representations of text.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA pretrained transformer-based language model used for language understanding and transfer learning.
• 

  liveBookmanning.com
  Build a Large Language Model (From Scratch)A transformer-based model built on the encoder stack and trained for bidirectional masked-word prediction.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA pretrained bidirectional Transformer encoder model for language understanding, used here as an NLP example rather than a generation model.

Compare how 9 books explain BERT →

Related concepts

• encoder-only transformerkind of · An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
• bidirectional language modelkind of · A bidirectional language model represents or predicts tokens using contextual information from both preceding and following tokens in a sequence.
• transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• masked language modelinguses · Masked language modeling is a self-supervised training objective in which a model predicts tokens hidden or replaced in an input sequence using surrounding context.
• DistilBERTincludes · DistilBERT is a smaller, faster Transformer language model distilled from BERT to retain much of its language-understanding capability with fewer parameters and lower resource requirements.

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.