9 Transfer Learning with Pretrained Language Models
This chapter covers:
- Using transfer learning to leverage knowledge from unlabeled textual data
- Using self-supervised learning to pretrain large language models such as BERT
- Building a sentiment analyzer with BERT and the HuggingFace Transformers library
- Building a natural language inference model with BERT and AllenNLP
The year of 2018 is often called “an inflection point” in the history of NLP. A prominent NLP researcher Sebastian Ruder[1] dubbed this change “NLP’s ImageNet moment,” where he used the name of a popular computer vision dataset and powerful models pretrained on it, pointing out the similar changes were underway in the NLP community as well. Powerful pretrained language models such as ELMo, BERT, and GPT-2 achieved state-of-the-art performance in many NLP tasks and completely changed how we build NLP models within months.
One important concept underlying these powerful pretrained language models is transfer learning. In this chapter, we’ll first introduce the concept, then move onto introducing BERT, the most popular pretrained language model proposed for NLP. We’ll cover how BERT is designed and pretrained, as well as how to use the model for downstream NLP tasks including sentiment analysis and natural language inference. We’ll also touch upon other popular pretrained models including ELMo and RoBERTa.