BERT
BERT is a pretrained Transformer encoder language model that uses bidirectional context to produce contextual representations of text.
Natural Language Processing in Action, Second Edition
BERT is a transformer language model pretrained with bidirectional contextual objectives for representing and understanding text.
- How it is used
- Deploys bidirectional representations for language understanding, sentence-pair tasks, question answering, and semantic search.
- How deep it goes
- Explains masking, tokenization, embeddings, pretraining, fine-tuning, and model sizes, while stopping short of exhaustive hardware guidance.
- Examples
- Uses masked words, sentence pairs, GLUE, SQuAD, PyTorch architecture summaries, and Haystack retrieval-generation pipelines.
- What you can do after
- Apply it via fine-tuning to classify sentence pairs, select answer spans, or encode text for semantic retrieval.
Read how Natural Language Processing in Action, Second Edition explains BERT →
LLMs in Production
A transformer-based language representation model that uses bidirectional context to learn text embeddings.
- How it is used
- Positions BERT as an architecture for training a task-specific model when customization and control matter.
- How deep it goes
- Names the architecture and illustrates its use, but does not explain its internals or provide training steps.
- Examples
- Uses Hugging Face Hub search results and a Slovak-language model trained on collected corpus data.
- What you can do after
- Evaluate/understand it as an option for training a focused model instead of relying on a general-purpose product.
AI-Powered Search
BERT is a transformer-based language model pretrained to produce contextual representations of text using bidirectional attention.
- How it is used
- Supports semantic search by producing contextual representations that can be compared as dense vectors.
- How deep it goes
- Explains masked-token pretraining, attention, embeddings, and sentence pooling, but defers full Transformer implementation details.
- Examples
- Uses masked sentences, weather phrases, vehicle phrases, 768-dimensional vectors, and a similarity matrix as teaching artifacts.
- What you can do after
- Apply it via SentenceTransformer to encode phrases, normalize embeddings, and rank pairs by similarity.
Machine Learning System Design
BERT is a pretrained bidirectional transformer language model that produces contextual representations of text.
- How it is used
- Uses pretrained BERT representations to convert sequential text into features for downstream machine-learning models.
- How deep it goes
- Defines BERT’s embedding role but stops short of explaining transformer internals, pretraining, or fine-tuning procedures.
- Examples
- Uses text features and user sessions of visited pages or watched videos as sequential embedding examples.
- What you can do after
- Apply BERT embeddings as low-dimensional input features for a final model.
Domain-Specific Small Language Models
A transformer-based language model that uses the encoder stack for understanding tasks such as classification.
- How it is used
- Uses encoder representations for text classification, prediction, and domain-specific fine-tuning.
- How deep it goes
- Explains architecture choice and fine-tuning preparation through tokenization, data splitting, loaders, and model loading, without training internals.
- Examples
- Demonstrates sentiment labels, bert-base-uncased tokenization, padded sequences, validation splits, data loaders, and sequence classification.
- What you can do after
- Builds a labeled-data pipeline and fine-tunes pretrained BERT for classification using tokenizers, PyTorch loaders, and Hugging Face models.
Read how Domain-Specific Small Language Models explains BERT →
Machine Learning Algorithms in Depth
BERT is a pretrained bidirectional transformer language model designed to learn contextual representations of text.
- How it is used
- Deploys pretrained bidirectional transformer representations across natural language processing tasks.
- How deep it goes
- Identifies BERT’s transfer-learning strategy and applications but does not derive its architecture or training procedure.
- Examples
- Uses question answering and language inference as concrete fine-tuning applications.
- What you can do after
- Apply it via fine-tuning with one additional output layer for downstream language tasks.
Read how Machine Learning Algorithms in Depth explains BERT →
Inside Deep Learning: Math, Algorithms, Models
A pretrained transformer-based language model used for language understanding and transfer learning.
- How it is used
- Applies pretrained transformer representations to NLP transfer learning with limited labeled data.
- How deep it goes
- Explains model selection, tokenization, batching, and computational tradeoffs, but stops short of BERT internals or training from scratch.
- Examples
- Uses AG News, a 256-item labeled split, a GRU comparison, DistilBERT, and Hugging Face batch encoding.
- What you can do after
- Apply it via Hugging Face Transformers to tokenize text and reuse a pretrained DistilBERT model for classification.
Read how Inside Deep Learning: Math, Algorithms, Models explains BERT →
Build a Large Language Model (From Scratch)
A transformer-based model built on the encoder stack and trained for bidirectional masked-word prediction.
- How it is used
- Uses masked-word prediction to support text classification tasks such as sentiment prediction and document categorization.
- How deep it goes
- Defines BERT’s encoder basis and training objective, while deferring self-attention implementation and input preprocessing details.
- Examples
- Uses masked sentences, sentiment prediction, document categorization, and toxic-content detection as concrete examples.
- What you can do after
- Apply it via text classification for sentiment, document categorization, or toxic-content detection.
Read how Build a Large Language Model (From Scratch) explains BERT →
Deep Learning with Python, Third Edition
A pretrained bidirectional Transformer encoder model for language understanding, used here as an NLP example rather than a generation model.
- How it is used
- Uses bidirectional attention to compute rich representations for language understanding rather than looping generation.
- How deep it goes
- Explains encoder directionality, masked-token training, unsupervised data collection, and contextual embeddings without detailing implementation internals.
- Examples
- Demonstrates masking roughly 15% of tokens, predicting replacements from surrounding text, and pretraining on Wikipedia.
- What you can do after
- Evaluate/understand when bidirectional masked-language modeling fits language-understanding tasks instead of autoregressive generation.
Read how Deep Learning with Python, Third Edition explains BERT →
Related concepts
- encoder-only transformerkind of · An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
- bidirectional language modelkind of · A bidirectional language model represents or predicts tokens using contextual information from both preceding and following tokens in a sequence.
- transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
- language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- masked language modelinguses · Masked language modeling is a self-supervised training objective in which a model predicts tokens hidden or replaced in an input sequence using surrounding context.
- DistilBERTincludes · DistilBERT is a smaller, faster Transformer language model distilled from BERT to retain much of its language-understanding capability with fewer parameters and lower resource requirements.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.