In this chapter and the previous chapter, our goal is to cover some representative deep transfer learning modeling architectures for natural language processing (NLP) that rely on a recently popularized neural architecture—the transformer1—for key functions. This is arguably the most important architecture for NLP today. Specifically, our goal has to look at modeling frameworks such as the generative pretrained transformer (GPT),2 Bidirectional Encoder Representations from Transformers (BERT),3 and multilingual BERT (mBERT).4 These methods employ neural networks with even more parameters than the deep convolutional and recurrent neural network models that we looked at previously. Despite their larger size, they have exploded in popularity because they scale comparatively more effectively on parallel computing architecture. This enables even larger and more sophisticated models to be developed in practice. To make the content more digestible, we split the coverage of these models into two chapters/parts: we covered the transformer and GPT neural network architectures in the previous chapter, and in this next chapter, we focus on BERT and mBERT.