BERT
BERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
Natural Language Processing in Action, Second Edition
BERT is a transformer language model pretrained with bidirectional contextual objectives for representing and understanding text.
- How it is used
- Deploys bidirectional representations for language understanding, sentence-pair tasks, question answering, and semantic search.
- How deep it goes
- Explains masking, tokenization, embeddings, pretraining, fine-tuning, and model sizes, while stopping short of exhaustive hardware guidance.
- Examples
- Uses masked words, sentence pairs, GLUE, SQuAD, PyTorch architecture summaries, and Haystack retrieval-generation pipelines.
- What you can do after
- Apply it via fine-tuning to classify sentence pairs, select answer spans, or encode text for semantic retrieval.
Read how Natural Language Processing in Action, Second Edition explains BERT →
LLMs in Production
A transformer-based language representation model that uses bidirectional context to learn text embeddings.
- How it is used
- Uses BERT to illustrate how custom model training can provide a competitive edge on specific tasks.
- How deep it goes
- Introduces BERT as a transformer architecture and describes a training project without detailing its internals or training procedure.
- Examples
- Describes SlovenBERTcina and its Slovak training corpus, and cites BERT models on the Hugging Face Hub.
- What you can do after
- Evaluate the potential advantage of training a task-specific model instead of relying on a general-purpose solution.
Retrieval Augmented Generation, The Foundational Ideas
BERT is a bidirectional Transformer-based language model pretrained to represent text using contextual token representations.
- How it is used
- Serves as a contextual reader for retrieved passages and a costly baseline for sentence-pair similarity.
- How deep it goes
- Explains bidirectional context, self-attention, pretraining, answer-span reading, and pairwise inference cost without detailing BERT implementation.
- Examples
- Uses ambiguous “bank,” support-ticket matching, a 10,000-sentence cost measurement, and a missed 401(k) answer passage.
- What you can do after
- Evaluate/understand when contextual reading helps and when pairwise cost or failed retrieval limits BERT.
Read how Retrieval Augmented Generation, The Foundational Ideas explains BERT →
Transfer Learning for Natural Language Processing
BERT (Bidirectional Encoder Representations from Transformers) is a transformer-based language model.
- How it is used
- Uses BERT as a pretrained transfer-learning baseline for NLP classification.
- How deep it goes
- Explains bidirectional encoder context and masked-word pretraining, then illustrates a Keras classification pipeline.
- Examples
- Uses email spam and IMDB movie-review classification to establish BERT baselines alongside traditional methods.
- What you can do after
- Apply BERT via TensorFlow Hub and Keras to construct a binary classifier with configurable fine-tuning.
Read how Transfer Learning for Natural Language Processing explains BERT →
AI-Powered Search
BERT is a transformer-based language model pretrained to produce contextual representations of text using bidirectional attention.
- How it is used
- Supports semantic search by producing contextual representations that can be compared as dense vectors.
- How deep it goes
- Explains masked-token pretraining, attention, embeddings, and sentence pooling, but defers full Transformer implementation details.
- Examples
- Uses masked sentences, weather phrases, vehicle phrases, 768-dimensional vectors, and a similarity matrix as teaching artifacts.
- What you can do after
- Apply it via SentenceTransformer to encode phrases, normalize embeddings, and rank pairs by similarity.
LLM Evaluation and Alignment, The Foundational Ideas
BERT is a bidirectional Transformer encoder model pretrained to produce contextual representations for natural-language understanding.
- How it is used
- Provides contextual representations for semantic evaluation and natural-language understanding.
- How deep it goes
- Explains contextual meaning, bidirectional attention, pooling, and cosine comparison, but defers BERT’s training procedure and architecture internals.
- Examples
- Uses bank disambiguation, agent-patient word order, legal-context encoding, pooling, and cosine similarity between paraphrased passages.
- What you can do after
- Apply it via Transformers to encode text and compare contextual representations with cosine similarity or BERTScore.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains BERT →
Build an Advanced RAG Application (From Scratch)
BERT is a bidirectional Transformer-based language model that produces contextual representations by attending to both sides of input tokens.
- How it is used
- Uses bidirectional contextual representations for downstream natural language processing tasks.
- How deep it goes
- Explains bidirectional context, masked language modeling, contextual embeddings, and sentence-pair differences, without implementation details.
- Examples
- Uses “The chef cooked [MASK] the meal” to contrast masked prediction with left-to-right language modeling.
- What you can do after
- Apply contextual representations to sentiment analysis, named entity recognition, and question answering.
Read how Build an Advanced RAG Application (From Scratch) explains BERT →
Machine Learning System Design
BERT is a pretrained bidirectional transformer language model that produces contextual representations of text.
- How it is used
- Uses pretrained BERT representations to convert sequential text into features for downstream machine-learning models.
- How deep it goes
- Defines BERT’s embedding role but stops short of explaining transformer internals, pretraining, or fine-tuning procedures.
- Examples
- Uses text features and user sessions of visited pages or watched videos as sequential embedding examples.
- What you can do after
- Apply BERT embeddings as low-dimensional input features for a final model.
Domain-Specific Small Language Models
A transformer-based language model that uses the encoder stack for understanding tasks such as classification.
- How it is used
- Handles text classification and prediction, including classification fine-tuned on domain-specific labeled data.
- How deep it goes
- Explains the encoder role and details classification-data preparation through tokenization, splitting, batching, and classifier loading.
- Examples
- Uses a positive/negative movie-review dataset and a bert-base-uncased sequence-classification setup.
- What you can do after
- Apply via Hugging Face Transformers, PyTorch, and scikit-learn to prepare labeled classification data and load a BERT classifier.
Read how Domain-Specific Small Language Models explains BERT →
Machine Learning Algorithms in Depth
BERT is a pretrained bidirectional transformer language model designed to learn contextual representations of text.
- How it is used
- Deploys pretrained bidirectional transformer representations across natural language processing tasks.
- How deep it goes
- Identifies BERT’s transfer-learning strategy and applications but does not derive its architecture or training procedure.
- Examples
- Uses question answering and language inference as concrete fine-tuning applications.
- What you can do after
- Apply it via fine-tuning with one additional output layer for downstream language tasks.
Read how Machine Learning Algorithms in Depth explains BERT →
Inside Deep Learning: Math, Algorithms, Models
A pretrained transformer-based language model used for language understanding and transfer learning.
- How it is used
- Uses pretrained transformer models to transfer language features to text classification with limited labeled data.
- How deep it goes
- Demonstrates loading a frozen model and tokenizer and encoding batches, without detailing BERT internals.
- Examples
- Compares a GRU baseline and frozen DistilBERT on AG News using 256 labeled examples.
- What you can do after
- Apply Hugging Face Transformers to load a pretrained DistilBERT model and encode text with its tokenizer.
Read how Inside Deep Learning: Math, Algorithms, Models explains BERT →
Build a Large Language Model (From Scratch)
A transformer-based model built on the encoder stack and trained for bidirectional masked-word prediction.
- How it is used
- Uses masked-word prediction to support text classification tasks such as sentiment prediction and document categorization.
- How deep it goes
- Defines BERT’s encoder basis and training objective, while deferring self-attention implementation and input preprocessing details.
- Examples
- Uses masked sentences, sentiment prediction, document categorization, and toxic-content detection as concrete examples.
- What you can do after
- Apply it via text classification for sentiment, document categorization, or toxic-content detection.
Read how Build a Large Language Model (From Scratch) explains BERT →
Deep Learning with Python, Third Edition
A pretrained bidirectional Transformer encoder model for language understanding, used here as an NLP example rather than a generation model.
- How it is used
- Uses bidirectional attention to compute rich representations for language understanding rather than looping generation.
- How deep it goes
- Explains encoder directionality, masked-token training, unsupervised data collection, and contextual embeddings without detailing implementation internals.
- Examples
- Demonstrates masking roughly 15% of tokens, predicting replacements from surrounding text, and pretraining on Wikipedia.
- What you can do after
- Evaluate/understand when bidirectional masked-language modeling fits language-understanding tasks instead of autoregressive generation.
Read how Deep Learning with Python, Third Edition explains BERT →
Also covered in
- Introduction to Generative AI, Second Edition
- Hugging Face in Action
- Deep Learning with R, Third Edition
- Data Without Labels
- AI Recommender SystemsIn early access (MEAP)
- Deep Learning with PyTorch, Second Edition
- Building Reliable AI Systems
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- natural language processingNatural language processing is the field of computing concerned with computational methods for analyzing, interpreting, transforming, and generating human language in text or speech.
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
Related concepts
- transformer architecturekind of · A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
- encoder-only transformerkind of · An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
- masked language modelinguses · Masked language modeling is a self-supervised training objective in which a model predicts original tokens hidden or replaced in an input sequence, using surrounding context.
- bidirectional language modelkind of · A bidirectional language model predicts or represents tokens using contextual information from both preceding and following tokens in a sequence.
- next sentence predictionuses · Next sentence prediction is a pretraining task that predicts whether one text segment follows another in the original text.
- Multilingual BERTincludes · Multilingual BERT (mBERT) is a BERT model pretrained on text in multiple languages to represent multilingual text.
Questions about BERT
What is BERT?
BERT (Bidirectional Encoder Representations from Transformers) is a pretrained Transformer encoder language model that produces contextual text representations using information from both directions.
Which Manning books explain BERT?
Natural Language Processing in Action, Second Edition, LLMs in Production, Retrieval Augmented Generation, The Foundational Ideas, Transfer Learning for Natural Language Processing, AI-Powered Search, LLM Evaluation and Alignment, The Foundational Ideas, Build an Advanced RAG Application (From Scratch), Machine Learning System Design and 5 more.
What should I know before learning BERT?
Manning books that teach BERT usually introduce language model, transformer architecture, natural language processing and embedding first.
What kinds of BERT are there?
Kinds of BERT include Multilingual BERT.
What is BERT a kind of?
BERT is a kind of transformer architecture. A transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
What is BERT also called?
BERT is also called Bidirectional Encoder Representations from Transformers.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.












