How deeply does LLMs in Production cover BERT?

Names the architecture and illustrates its use, but does not explain its internals or provide training steps.

BERT

A transformer-based language representation model that uses bidirectional context to learn text embeddings.

If a general-purpose model is not the best fit for your data, BERT offers another path. Bidirectional Encoder Representations from Transformers is a transformer model architecture you can use to train your own model for a particular task. section 1.2.2

That choice is about control as well as performance: owning a model gives you room to customize it rather than relying entirely on someone else’s product. It also means doing the training yourself. A search for “BERT” on the Hugging Face Hub returned more than 13,700 individually trained models at the time cited—evidence of how differently people have put the architecture to work. section 1.2.2

A task-specific model

SlovenBERTcina was trained using a large Slovak-language dataset assembled from several sources, including the Slovak National Corpus, OSCAR, and Europarl. It did not set computational records, but it outperformed other available models on the tasks it was trained for. The practical question is whether a focused model can solve your task better—not whether it is the biggest model. section 1.2.2

How LLMs in Production covers it

liveBook manning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Natural Language Processing in Action, Second EditionBERT is a transformer language model pretrained with bidirectional contextual objectives for representing and understanding text.
liveBook manning.com

AI-Powered SearchBERT is a transformer-based language model pretrained to produce contextual representations of text using bidirectional attention.
liveBook manning.com

Machine Learning System DesignBERT is a pretrained bidirectional transformer language model that produces contextual representations of text.
liveBook manning.com

Domain-Specific Small Language ModelsA transformer-based language model that uses the encoder stack for understanding tasks such as classification.
liveBook manning.com

Machine Learning Algorithms in DepthBERT is a pretrained bidirectional transformer language model designed to learn contextual representations of text.
liveBook manning.com

Inside Deep Learning: Math, Algorithms, ModelsA pretrained transformer-based language model used for language understanding and transfer learning.
liveBook manning.com

Build a Large Language Model (From Scratch)A transformer-based model built on the encoder stack and trained for bidirectional masked-word prediction.
liveBook manning.com

Deep Learning with Python, Third EditionA pretrained bidirectional Transformer encoder model for language understanding, used here as an NLP example rather than a generation model.

Compare how 9 books explain BERT →

Related concepts

encoder-only transformerkind of · An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
bidirectional language modelkind of · A bidirectional language model represents or predicts tokens using contextual information from both preceding and following tokens in a sequence.
transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
masked language modelinguses · Masked language modeling is a self-supervised training objective in which a model predicts tokens hidden or replaced in an input sequence using surrounding context.
DistilBERTincludes · DistilBERT is a smaller, faster Transformer language model distilled from BERT to retain much of its language-understanding capability with fewer parameters and lower resource requirements.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.