11 Conclusions

 

This chapter covers

  • Summarizing important concepts covered by this book
  • Summarizing related important emerging concepts
  • Considering limitations and environmental and ethical considerations around transfer learning methods for NLP
  • Envisioning the future of transfer learning in NLP
  • Keeping up with latest developments in the field

We’ve covered a great deal of material in the preceding chapters—we hope they were informative and engaging. This concluding chapter attempts to provide a meaningful summary of everything we did and looks to the future of the field and emerging research trends. Due to the prolific output and the quick-moving nature of the field, we certainly have not covered every single influential architecture or promising research direction. To mitigate this, we include a brief discussion of various research trends we did not get a chance to cover in this book, making connections to, and framing in the context of, covered material as much as possible.

In this chapter, we also try to provide a broader context by touching on emerging questions that have not traditionally been given much attention, such as ethical considerations and the environmental impact of the various models. These are closely tied to the awareness of the limitations of these models, which we try to highlight as much as possible in this chapter.

11.1 Overview of key concepts

11.2 Other emerging research trends

11.2.1 RoBERTa

11.2.2 GPT-3

11.2.3 XLNet

11.2.4 BigBird

11.2.5 Longformer

11.2.6 Reformer

11.2.7 T5

11.2.8 BART

11.2.9 XLM

11.2.10 TAPAS

11.3 Future of transfer learning in NLP

11.4 Ethical and environmental considerations

11.5 Staying up-to-date

11.5.1 Kaggle and Zindi competitions

11.5.2 arXiv

11.5.3 News and social media (Twitter)