chapter thirteen

13 Transformers

 

This chapter covers,

  • Implementing a transformer model that has all the components as discussed in the original Transformer paper including positional embeddings and layer normalization
  • Implementing BERT and a downstream classifier to identify spam messages using TensorFlow and TFHub
  • Implementing a question answering model based on BERT using HuggingFace’s transformer library

13.1 Transformers in more detail

13.1.1 Revisiting the basic components of the transformer

13.1.2 Embeddings in the transformer

13.1.3 Residuals and normalization

13.2 Using pre-trained BERT for spam classification

13.2.1 Understanding BERT

13.2.2 Classifying spam with BERT in TensorFlow

13.3 Question answering with HuggingFace’s transformers

13.3.1 Understanding the data

13.3.2 Processing data

13.3.3 Defining the DistilBERT model

13.3.4 Training the model

13.3.5 Ask BERT a question

13.4 Summary