liveBook · Manning

What you will learn in Transfer Learning for Natural Language Processing

by Paul Azunre

Chapter by chapter

Chapter 1: What is transfer learning?

This introduction defines transfer learning and explains how reusing prior knowledge and pretrained models can make NLP more accessible.

1.1 Overview of representative NLP tasks · 1.2 Understanding NLP in the context of AI · 1.2.1 Artificial intelligence (AI) · 1.2.2 Machine learning · 1.2.3 Natural language processing (NLP) · 1.3 A brief history of NLP advances · 1.3.1 General overview · 1.3.2 Recent transfer learning advances · 1.4 Transfer learning in computer vision · 1.4.1 General overview · 1.4.2 Pretrained ImageNet models · 1.4.3 Fine-tuning pretrained ImageNet models · 1.5 Why is NLP transfer learning an exciting topic to study now? · 1.6 Summary

What you learn in chapter 1 →

Chapter 2: Getting started with baselines: Data preprocessing

This introduction explains the purpose of NLP baselines, the limitations of keyword rules, and how to run the book’s notebooks with attention to dependencies.

2.1 Preprocessing email spam classification example data · 2.1.1 Loading and visualizing the Enron corpus · 2.1.2 Loading and visualizing the fraudulent email corpus · 2.1.3 Converting the email text into numbers · 2.2 Preprocessing movie sentiment classification example data · 2.3 Generalized linear models · 2.3.1 Logistic regression · 2.3.2 Support vector machines (SVMs) · 2.4 Summary

What you learn in chapter 2 →

Chapter 3: Getting started with baselines: Benchmarking and optimization

This introduction frames baselines for two NLP classification problems as points of comparison and describes a simple pretrained-model transfer approach.

3.1 Decision-tree-based models · 3.1.1 Random forests (RFs) · 3.1.2 Gradient-boosting machines (GBMs) · 3.2 Neural network models · 3.2.1 Embeddings from Language Models (ELMo) · 3.2.2 Bidirectional Encoder Representations from Transformers (BERT) · 3.3 Optimizing performance · 3.3.1 Manual hyperparameter tuning · 3.3.2 Systematic hyperparameter tuning · 3.4 Summary

What you learn in chapter 3 →

Chapter 4: Shallow transfer learning for NLP

The introduction frames shallow transfer learning through pretrained embeddings, multitask learning, and domain adaptation.

4.1 Semisupervised learning with pretrained word embeddings · 4.2 Semisupervised learning with higher-level representations · 4.3 Multitask learning · 4.3.1 Problem setup and a shallow neural single-task baseline · 4.3.2 Dual-task experiment · 4.4 Domain adaptation · 4.5 Summary

What you learn in chapter 4 →

Chapter 5: Preprocessing data for recurrent neural network deep transfer learning experiments

The chapter introduction contrasts contextual and static embeddings, outlines SIMOn and ELMo, and introduces two ULMFiT adaptation techniques.

5.1 Preprocessing tabular column-type classification data · 5.1.1 Obtaining and visualizing tabular data · 5.1.2 Preprocessing tabular data · 5.1.3 Encoding preprocessed data as numbers · 5.2 Preprocessing fact-checking example data · 5.2.1 Special problem considerations · 5.2.2 Loading and visualizing fact-checking data · 5.3 Summary

What you learn in chapter 5 →

Chapter 6: Deep transfer learning for NLP with recurrent neural networks

This introduction previews RNN-based transfer learning methods and their application to column-type classification and fake news detection.

6.1 Semantic Inference for the Modeling of Ontologies (SIMOn) · 6.1.1 General neural architecture overview · 6.1.2 Modeling tabular data · 6.1.3 Application of SIMOn to tabular column-type classification data · 6.2 Embeddings from Language Models (ELMo) · 6.2.1 ELMo bidirectional language modeling · 6.2.2 Application to fake news detection · 6.3 Universal Language Model Fine-Tuning (ULMFiT) · 6.3.1 Target task language model fine-tuning · 6.3.2 Target task classifier fine-tuning · 6.4 Summary

What you learn in chapter 6 →

Chapter 7: Deep transfer learning for NLP with the transformer and GPT

This introduction contrasts transformer attention and scaling with earlier NLP architectures and summarizes GPT, BERT, and multilingual BERT.

7.1 The transformer · 7.1.1 An introduction to the transformers library and attention visualization · 7.1.2 Self-attention · 7.1.3 Residual connections, encoder-decoder attention, and positional encoding · 7.1.4 Application of pretrained encoder-decoder to translation · 7.2 The Generative Pretrained Transformer · 7.2.1 Architecture overview · 7.2.2 Transformers pipelines introduction and application to text generation · 7.2.3 Application to chatbots · 7.3 Summary

What you learn in chapter 7 →

Chapter 8: Deep transfer learning for NLP with BERT and multilingual BERT

This introduction summarizes transformer scaling, BERT’s training objectives, and multilingual BERT’s suitability for cross-lingual transfer.

8.1 Bidirectional Encoder Representations from Transformers (BERT) · 8.1.1 Model architecture · 8.1.2 Application to question answering · 8.1.3 Application to fill in the blanks and next-sentence prediction tasks · 8.2 Cross-lingual learning with multilingual BERT (mBERT) · 8.2.1 Brief JW300 dataset overview · 8.2.2 Transfer mBERT to monolingual Twi data with the pretrained tokenizer · 8.2.3 mBERT and tokenizer trained from scratch on monolingual Twi data · 8.3 Summary

What you learn in chapter 8 →

Chapter 9: ULMFiT and knowledge distillation adaptation strategies

This introduction explains parameter-efficient transfer learning and briefly describes ULMFiT fine-tuning strategies and knowledge distillation.

9.1 Gradual unfreezing and discriminative fine-tuning · 9.1.1 Pretrained language model fine-tuning · 9.1.2 Target task classifier fine-tuning · 9.2 Knowledge distillation · 9.2.1 Transfer DistilmBERT to monolingual Twi data with pretrained tokenizer · 9.3 Summary

What you learn in chapter 9 →

Chapter 10: ALBERT, adapters, and multitask adaptation strategies

This introduction explains several strategies for making pretrained language models more scalable or efficient to adapt.

10.1 Embedding factorization and cross-layer parameter sharing · 10.1.1 Fine-tuning pretrained ALBERT on MDSD book reviews · 10.2 Multitask fine-tuning · 10.2.1 General Language Understanding Dataset (GLUE) · 10.2.2 Fine-tuning on a single GLUE task · 10.2.3 Sequential adaptation · 10.3 Adapters · 10.4 Summary

What you learn in chapter 10 →

Chapter 11: Conclusions

The introduction frames the concluding chapter and emphasizes continued learning in a rapidly changing NLP field.

11.1 Overview of key concepts · 11.2 Other emerging research trends · 11.2.1 RoBERTa · 11.2.2 GPT-3 · 11.2.3 XLNet · 11.2.4 BigBird · 11.2.5 Longformer · 11.2.6 Reformer · 11.2.7 T5 · 11.2.8 BART · 11.2.9 XLM · 11.2.10 TAPAS · 11.3 Future of transfer learning in NLP · 11.4 Ethical and environmental considerations · 11.5 Staying up-to-date · 11.5.1 Kaggle and Zindi competitions · 11.5.2 arXiv · 11.5.3 News and social media (Twitter) · 11.6 Final words · 11.7 Summary

What you learn in chapter 11 →

Concepts explained in this book

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.