liveBook · Manning

BERT

Also called Bidirectional Encoder Representations from Transformers

BERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.

How 13 Manning books explain it

Natural Language Processing in Action, Second Edition

BERT is a transformer language model pretrained with bidirectional contextual objectives for representing and understanding text.

How it is used
Deploys bidirectional representations for language understanding, sentence-pair tasks, question answering, and semantic search.
How deep it goes
Explains masking, tokenization, embeddings, pretraining, fine-tuning, and model sizes, while stopping short of exhaustive hardware guidance.
Examples
Uses masked words, sentence pairs, GLUE, SQuAD, PyTorch architecture summaries, and Haystack retrieval-generation pipelines.
What you can do after
Apply it via fine-tuning to classify sentence pairs, select answer spans, or encode text for semantic retrieval.

Read how Natural Language Processing in Action, Second Edition explains BERT →

LLMs in Production

A transformer-based language representation model that uses bidirectional context to learn text embeddings.

How it is used
Uses BERT to illustrate how custom model training can provide a competitive edge on specific tasks.
How deep it goes
Introduces BERT as a transformer architecture and describes a training project without detailing its internals or training procedure.
Examples
Describes SlovenBERTcina and its Slovak training corpus, and cites BERT models on the Hugging Face Hub.
What you can do after
Evaluate the potential advantage of training a task-specific model instead of relying on a general-purpose solution.

Read how LLMs in Production explains BERT →

Retrieval Augmented Generation, The Foundational Ideas

BERT is a bidirectional Transformer-based language model pretrained to represent text using contextual token representations.

How it is used
Serves as a contextual reader for retrieved passages and a costly baseline for sentence-pair similarity.
How deep it goes
Explains bidirectional context, self-attention, pretraining, answer-span reading, and pairwise inference cost without detailing BERT implementation.
Examples
Uses ambiguous “bank,” support-ticket matching, a 10,000-sentence cost measurement, and a missed 401(k) answer passage.
What you can do after
Evaluate/understand when contextual reading helps and when pairwise cost or failed retrieval limits BERT.

Read how Retrieval Augmented Generation, The Foundational Ideas explains BERT →

Transfer Learning for Natural Language Processing

BERT (Bidirectional Encoder Representations from Transformers) is a transformer-based language model.

How it is used
Uses BERT as a pretrained transfer-learning baseline for NLP classification.
How deep it goes
Explains bidirectional encoder context and masked-word pretraining, then illustrates a Keras classification pipeline.
Examples
Uses email spam and IMDB movie-review classification to establish BERT baselines alongside traditional methods.
What you can do after
Apply BERT via TensorFlow Hub and Keras to construct a binary classifier with configurable fine-tuning.

Read how Transfer Learning for Natural Language Processing explains BERT →

AI-Powered Search

BERT is a transformer-based language model pretrained to produce contextual representations of text using bidirectional attention.

How it is used
Supports semantic search by producing contextual representations that can be compared as dense vectors.
How deep it goes
Explains masked-token pretraining, attention, embeddings, and sentence pooling, but defers full Transformer implementation details.
Examples
Uses masked sentences, weather phrases, vehicle phrases, 768-dimensional vectors, and a similarity matrix as teaching artifacts.
What you can do after
Apply it via SentenceTransformer to encode phrases, normalize embeddings, and rank pairs by similarity.

Read how AI-Powered Search explains BERT →

LLM Evaluation and Alignment, The Foundational Ideas

BERT is a bidirectional Transformer encoder model pretrained to produce contextual representations for natural-language understanding.

How it is used
Provides contextual representations for semantic evaluation and natural-language understanding.
How deep it goes
Explains contextual meaning, bidirectional attention, pooling, and cosine comparison, but defers BERT’s training procedure and architecture internals.
Examples
Uses bank disambiguation, agent-patient word order, legal-context encoding, pooling, and cosine similarity between paraphrased passages.
What you can do after
Apply it via Transformers to encode text and compare contextual representations with cosine similarity or BERTScore.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains BERT →

Build an Advanced RAG Application (From Scratch)

BERT is a bidirectional Transformer-based language model that produces contextual representations by attending to both sides of input tokens.

How it is used
Uses bidirectional contextual representations for downstream natural language processing tasks.
How deep it goes
Explains bidirectional context, masked language modeling, contextual embeddings, and sentence-pair differences, without implementation details.
Examples
Uses “The chef cooked [MASK] the meal” to contrast masked prediction with left-to-right language modeling.
What you can do after
Apply contextual representations to sentiment analysis, named entity recognition, and question answering.

Read how Build an Advanced RAG Application (From Scratch) explains BERT →

Machine Learning System Design

BERT is a pretrained bidirectional transformer language model that produces contextual representations of text.

How it is used
Uses pretrained BERT representations to convert sequential text into features for downstream machine-learning models.
How deep it goes
Defines BERT’s embedding role but stops short of explaining transformer internals, pretraining, or fine-tuning procedures.
Examples
Uses text features and user sessions of visited pages or watched videos as sequential embedding examples.
What you can do after
Apply BERT embeddings as low-dimensional input features for a final model.

Read how Machine Learning System Design explains BERT →

Domain-Specific Small Language Models

A transformer-based language model that uses the encoder stack for understanding tasks such as classification.

How it is used
Handles text classification and prediction, including classification fine-tuned on domain-specific labeled data.
How deep it goes
Explains the encoder role and details classification-data preparation through tokenization, splitting, batching, and classifier loading.
Examples
Uses a positive/negative movie-review dataset and a bert-base-uncased sequence-classification setup.
What you can do after
Apply via Hugging Face Transformers, PyTorch, and scikit-learn to prepare labeled classification data and load a BERT classifier.

Read how Domain-Specific Small Language Models explains BERT →

Machine Learning Algorithms in Depth

BERT is a pretrained bidirectional transformer language model designed to learn contextual representations of text.

How it is used
Deploys pretrained bidirectional transformer representations across natural language processing tasks.
How deep it goes
Identifies BERT’s transfer-learning strategy and applications but does not derive its architecture or training procedure.
Examples
Uses question answering and language inference as concrete fine-tuning applications.
What you can do after
Apply it via fine-tuning with one additional output layer for downstream language tasks.

Read how Machine Learning Algorithms in Depth explains BERT →

Inside Deep Learning: Math, Algorithms, Models

A pretrained transformer-based language model used for language understanding and transfer learning.

How it is used
Uses pretrained transformer models to transfer language features to text classification with limited labeled data.
How deep it goes
Demonstrates loading a frozen model and tokenizer and encoding batches, without detailing BERT internals.
Examples
Compares a GRU baseline and frozen DistilBERT on AG News using 256 labeled examples.
What you can do after
Apply Hugging Face Transformers to load a pretrained DistilBERT model and encode text with its tokenizer.

Read how Inside Deep Learning: Math, Algorithms, Models explains BERT →

Build a Large Language Model (From Scratch)

A transformer-based model built on the encoder stack and trained for bidirectional masked-word prediction.

How it is used
Uses masked-word prediction to support text classification tasks such as sentiment prediction and document categorization.
How deep it goes
Defines BERT’s encoder basis and training objective, while deferring self-attention implementation and input preprocessing details.
Examples
Uses masked sentences, sentiment prediction, document categorization, and toxic-content detection as concrete examples.
What you can do after
Apply it via text classification for sentiment, document categorization, or toxic-content detection.

Read how Build a Large Language Model (From Scratch) explains BERT →

Deep Learning with Python, Third Edition

A pretrained bidirectional Transformer encoder model for language understanding, used here as an NLP example rather than a generation model.

How it is used
Uses bidirectional attention to compute rich representations for language understanding rather than looping generation.
How deep it goes
Explains encoder directionality, masked-token training, unsupervised data collection, and contextual embeddings without detailing implementation internals.
Examples
Demonstrates masking roughly 15% of tokens, predicting replacements from surrounding text, and pretraining on Wikipedia.
What you can do after
Evaluate/understand when bidirectional masked-language modeling fits language-understanding tasks instead of autoregressive generation.

Read how Deep Learning with Python, Third Edition explains BERT →

Also covered in

What to know first

Related concepts

Questions about BERT

What is BERT?

BERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.

Which Manning books explain BERT?

Natural Language Processing in Action, Second Edition, LLMs in Production, Retrieval Augmented Generation, The Foundational Ideas, Transfer Learning for Natural Language Processing, AI-Powered Search, LLM Evaluation and Alignment, The Foundational Ideas, Build an Advanced RAG Application (From Scratch), Machine Learning System Design and 5 more.

What should I know before learning BERT?

Manning books that teach BERT usually introduce language model, transformer architecture, natural language processing and embedding first.

What kinds of BERT are there?

Kinds of BERT include Multilingual BERT.

What is BERT a kind of?

BERT is a kind of transformer architecture. A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.

What is BERT also called?

BERT is also called Bidirectional Encoder Representations from Transformers.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.