4 Shallow transfer learning for NLP

 

This chapter covers

  • Using pretrained word embeddings in a semisupervised fashion to transfer pretrained knowledge to a problem
  • Using pretrained embeddings of larger sections of text in a semisupervised fashion to transfer pretrained knowledge to a problem
  • Using multitask learning to develop better-performing models
  • Modifying target domain data to reuse knowledge from a resource-rich source domain

In this chapter, we will cover some prominent shallow transfer learning approaches and concepts. This allows us to explore some major themes in transfer learning, while doing so within the context of relatively simple models in the class of eventual interest—shallow neural networks. Several authors have suggested various classification systems for categorizing transfer learning methods into groups.1,2,3 Roughly speaking, categorization is based on whether transfer occurs between different languages, tasks, or data domains. Each of these types of categorization is usually correspondingly referred to as cross-lingual learning, multitask learning, and domain adaptation, as visualized in figure 4.1.

Figure 4.1 Visualizing the categorization of transfer learning into multitask learning, domain adaptation, and cross-lingual learning
04_01

4.1 Semisupervised learning with pretrained word embeddings

4.2 Semisupervised learning with higher-level representations

4.3 Multitask learning

4.3.1 Problem setup and a shallow neural single-task baseline

4.3.2 Dual-task experiment

4.4 Domain adaptation

Summary