How deeply does Machine Learning Algorithms in Depth cover BERT?

Identifies BERT’s transfer-learning strategy and applications but does not derive its architecture or training procedure.

BERT

As explained in Machine Learning Algorithms in Depth

BERT is a pretrained bidirectional transformer language model designed to learn contextual representations of text.

BERT is a pretrained bidirectional transformer language model that learns contextual representations of text for natural language processing tasks. section 11.5

Its practical role is transfer learning: fine-tune the pretrained model with one additional output layer instead of designing substantial task-specific architectures. section 11.5

BERT supports tasks including:

Question answering. section 11.5
Language inference. section 11.5

Task adaptation

Start with pretrained BERT, attach one output layer, and fine-tune the resulting model for question answering or language inference. section 11.5

How Machine Learning Algorithms in Depth covers it

Cover of Machine Learning Algorithms in Depth

liveBook manning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Natural Language Processing in Action, Second EditionBERT is a transformer language model pretrained with bidirectional contextual objectives for representing and understanding text.
liveBook manning.com

LLMs in ProductionA transformer-based language representation model that uses bidirectional context to learn text embeddings.
liveBook manning.com

AI-Powered SearchBERT is a transformer-based language model pretrained to produce contextual representations of text using bidirectional attention.
liveBook manning.com

Machine Learning System DesignBERT is a pretrained bidirectional transformer language model that produces contextual representations of text.
liveBook manning.com

Domain-Specific Small Language ModelsA transformer-based language model that uses the encoder stack for understanding tasks such as classification.
liveBook manning.com

Inside Deep Learning: Math, Algorithms, ModelsA pretrained transformer-based language model used for language understanding and transfer learning.
liveBook manning.com

Build a Large Language Model (From Scratch)A transformer-based model built on the encoder stack and trained for bidirectional masked-word prediction.
liveBook manning.com

Deep Learning with Python, Third EditionA pretrained bidirectional Transformer encoder model for language understanding, used here as an NLP example rather than a generation model.

Compare how 9 books explain BERT →

Related concepts

encoder-only transformerkind of · An encoder-only Transformer is a Transformer model composed of encoder layers that process an input sequence into contextual representations without a separate decoder.
bidirectional language modelkind of · A bidirectional language model represents or predicts tokens using contextual information from both preceding and following tokens in a sequence.
transformer architecturekind of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
language modelkind of · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
masked language modelinguses · Masked language modeling is a self-supervised training objective in which a model predicts tokens hidden or replaced in an input sequence using surrounding context.
DistilBERTincludes · DistilBERT is a smaller, faster Transformer language model distilled from BERT to retain much of its language-understanding capability with fewer parameters and lower resource requirements.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.