How deeply does Deep Learning with Python, Third Edition cover BERT?

Explains encoder directionality, masked-token training, unsupervised data collection, and contextual embeddings without detailing implementation internals.

BERT

As explained in Deep Learning with Python, Third Edition

A pretrained bidirectional Transformer encoder model for language understanding, used here as an NLP example rather than a generation model.

BERT is a pretrained bidirectional Transformer encoder for building rich representations of input text. Every position can attend to tokens before and after it, making BERT useful for language understanding rather than generation in a loop. section 15.4.1

Masked language modeling

To pretrain BERT, replace roughly 15% of tokens in a text sequence with [MASK]. The model predicts the original tokens from their surrounding context. This unsupervised setup requires no labels and can use large text collections such as Wikipedia. section 15.4.1

Keep these distinctions in mind:

BERT predicts masked tokens from both directions; a causal language model predicts each token from preceding tokens. section 15.4.1
Pretraining the entire Transformer produces context-sensitive word embeddings, rather than fixed embeddings independent of surrounding words. section 15.4.1

How Deep Learning with Python, Third Edition covers it

Cover of Deep Learning with Python, Third Edition

liveBook manning.com

Read Deep Learning with Python, Third Edition on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Natural Language Processing in Action, Second EditionBERT is a transformer language model pretrained with bidirectional contextual objectives for representing and understanding text.
liveBook manning.com

LLMs in ProductionA transformer-based language representation model that uses bidirectional context to learn text embeddings.
liveBook manning.com

AI-Powered SearchBERT is a transformer-based language model pretrained to produce contextual representations of text using bidirectional attention.
liveBook manning.com

Machine Learning System DesignBERT is a pretrained bidirectional transformer language model that produces contextual representations of text.
liveBook manning.com

Domain-Specific Small Language ModelsA transformer-based language model that uses the encoder stack for understanding tasks such as classification.
liveBook manning.com

Machine Learning Algorithms in DepthBERT is a pretrained bidirectional transformer language model designed to learn contextual representations of text.
liveBook manning.com

Inside Deep Learning: Math, Algorithms, ModelsA pretrained transformer-based language model used for language understanding and transfer learning.
liveBook manning.com

Build a Large Language Model (From Scratch)A transformer-based model built on the encoder stack and trained for bidirectional masked-word prediction.

Compare how 9 books explain BERT →

Related concepts

encoder-only transformerkind of · An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
bidirectional language modelkind of · A bidirectional language model represents or predicts tokens using contextual information from both preceding and following tokens in a sequence.
transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
masked language modelinguses · Masked language modeling is a self-supervised training objective in which a model predicts tokens hidden or replaced in an input sequence using surrounding context.
DistilBERTincludes · DistilBERT is a smaller, faster Transformer language model distilled from BERT to retain much of its language-understanding capability with fewer parameters and lower resource requirements.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with Python, Third Edition. The section links point to the parts of the book it draws on.