What you will learn in Transfer Learning for Natural Language Processing
Chapter by chapter
Chapter 1: What is transfer learning?
This introduction defines transfer learning and explains how reusing prior knowledge and pretrained models can make NLP more accessible.
What you learn in chapter 1 →Chapter 2: Getting started with baselines: Data preprocessing
This introduction explains the purpose of NLP baselines, the limitations of keyword rules, and how to run the book’s notebooks with attention to dependencies.
What you learn in chapter 2 →Chapter 3: Getting started with baselines: Benchmarking and optimization
This introduction frames baselines for two NLP classification problems as points of comparison and describes a simple pretrained-model transfer approach.
What you learn in chapter 3 →Chapter 4: Shallow transfer learning for NLP
The introduction frames shallow transfer learning through pretrained embeddings, multitask learning, and domain adaptation.
What you learn in chapter 4 →Chapter 5: Preprocessing data for recurrent neural network deep transfer learning experiments
The chapter introduction contrasts contextual and static embeddings, outlines SIMOn and ELMo, and introduces two ULMFiT adaptation techniques.
What you learn in chapter 5 →Chapter 6: Deep transfer learning for NLP with recurrent neural networks
This introduction previews RNN-based transfer learning methods and their application to column-type classification and fake news detection.
What you learn in chapter 6 →Chapter 7: Deep transfer learning for NLP with the transformer and GPT
This introduction contrasts transformer attention and scaling with earlier NLP architectures and summarizes GPT, BERT, and multilingual BERT.
What you learn in chapter 7 →Chapter 8: Deep transfer learning for NLP with BERT and multilingual BERT
This introduction summarizes transformer scaling, BERT’s training objectives, and multilingual BERT’s suitability for cross-lingual transfer.
What you learn in chapter 8 →Chapter 9: ULMFiT and knowledge distillation adaptation strategies
This introduction explains parameter-efficient transfer learning and briefly describes ULMFiT fine-tuning strategies and knowledge distillation.
What you learn in chapter 9 →Chapter 10: ALBERT, adapters, and multitask adaptation strategies
This introduction explains several strategies for making pretrained language models more scalable or efficient to adapt.
What you learn in chapter 10 →Chapter 11: Conclusions
The introduction frames the concluding chapter and emphasizes continued learning in a rapidly changing NLP field.
What you learn in chapter 11 →Concepts explained in this book
- transfer learningTransfer learning is a machine-learning paradigm in which knowledge acquired from one task or domain is adapted to another task or domain.
- fine-tuningFine-tuning is adapting a pretrained model to a new task by updating some or all of its parameters.
- progressive unfreezingGradual unfreezing is a fine-tuning procedure that progressively makes more model parameters trainable.
- column-type classificationColumn-type classification is the task of assigning a data column to a category based on its type.
- ELMoELMo is a deep contextualized word-representation model that generates context-dependent representations from a bidirectional language model.
- natural language processingNatural language processing is the field concerned with computationally interpreting, understanding, processing, and acting on human language.
- preprocessingData preprocessing is the preparation and transformation of raw data into a form suitable for analysis or machine learning.
- transformer architectureA transformer is a neural network architecture that models sequences using attention mechanisms rather than recurrent connections.
- sequential adaptationSequential adaptation is a transfer-learning strategy that divides an overall adaptation into steps performed one after another.
- ULMFiTULMFiT is a transfer-learning method that fine-tunes a pretrained neural language model for a target task.
- deep learningDeep learning is a machine-learning approach that uses neural networks with multiple layers to learn representations from data.
- self-attentionSelf-attention is an attention mechanism that relates positions within a sequence to other positions in that sequence.
- distillationKnowledge distillation is a neural-network compression technique that trains a smaller student model to mimic the outputs of a larger teacher model.
- adapterAn adapter is a small trainable neural-network module inserted between layers of a pretrained network while its original weights remain fixed.
- zero-shot transfer learningTransfer learning in which a model is applied to a target setting without fine-tuning on data from that setting.
- discriminative fine-tuningDiscriminative fine-tuning trains different layers of a network at different learning rates.
- GPT modelGPT is a family of autoregressive transformer language models developed by OpenAI.
- fake news detectionThe example task of detecting false or misleading information presented in the form of news.
- BERTBERT (Bidirectional Encoder Representations from Transformers) is a transformer-based language model.
- multitask trainingA learning technique that trains an algorithm to perform well on multiple tasks simultaneously.
- cross-lingual transfer learningA transfer-learning approach that applies knowledge learned from one language to a task involving another language.
- autoencoderA model that learns to reconstruct its input, typically by encoding it into a latent representation and decoding that representation.
- domain adaptationA transfer-learning technique that adapts data or learned representations from a source domain to support learning or inference in a different target domain.
- ALBERTALBERT is a parameter-efficient language-model architecture based on BERT that reduces model size through factorized embeddings and parameter sharing.
- neural networkA neural network is a computational model composed of interconnected units whose parameters are adjusted to learn patterns from data.
- gradient boostingAn ensemble method that iteratively adds decision-tree-based models to address weaknesses in earlier models.
- pretrained embeddingA pretrained embedding is a learned vector representation of linguistic units produced before being applied to a downstream task.
- parameter efficiencyParameter efficiency is the goal of reducing a model’s number of parameters while preserving as much performance as possible.
- multitask fine-tuningMultitask fine-tuning adapts a pretrained model by training it on multiple tasks, encouraging knowledge sharing among them.
- masked language modelingMasked language modeling is a training objective that predicts words hidden or masked in an input sequence.
- machine learningA field of computing in which algorithms learn patterns or transformations from data rather than being explicitly programmed for every scenario.
- DistilBERTA smaller transformer language model trained to retain much of BERT’s language understanding capabilities through knowledge distillation.
- text generationText generation is the process of producing a sequence of text, often by predicting successive tokens.
- next sentence predictionNext sentence prediction is a pretraining task that predicts whether one text segment follows another in the original text.
- feature engineeringFeature engineering is the process of transforming input data into features suitable for machine-learning methods.
- machine translationMachine translation is the automatic translation of text or speech from one natural language to another.
- zero-shot domain adaptationDomain adaptation that transfers knowledge to a target domain without labeled data from that domain.
- embedding factorizationEmbedding factorization represents a vocabulary-to-hidden-layer projection using two matrices separated by a smaller intermediate embedding dimension.
- contextualized representationA vector representation of a linguistic item whose values depend on its surrounding context.
- embeddingAn embedding is a vector representation of an item in a space where relationships among items can be modeled geometrically.
- positional embeddingA method of representing sequence position by adding position-dependent information to token representations.
- embedding averagingA technique that represents a larger text by taking the mean of its word-embedding vectors.
- FastTextA word-representation method that applies Skip-Gram to character n-grams to represent words, including unseen words.
- hypothesis setA predefined set of candidate functions from which a learning algorithm selects a transformation.
- decision boundaryA function or surface that separates input examples assigned to different classes.
- word embeddingA representation of a word as a vector of real numbers, or a technique for learning such representations.
- biasBias is systematic skew in judgments or outcomes, including skew introduced when people prepare training labels.
- hyperparameter tuningThe process of searching for hyperparameter values that improve a model's performance.
- hard parameter sharingA multitask-learning pattern in which tasks share model parameters while retaining task-specific components.
- recurrent neural networkA recurrent neural network is a neural network that processes sequential data while maintaining state across sequence positions.
- SIMOnSIMOn is a neural model for structural semantic text classification that combines character-level convolutional networks with bidirectional language-model representations.
- generalized linear modelA statistical model in which a function links a linear combination of independent variables to the expected value of an output variable.
- shallow transfer learningTransfer learning that reuses representations or knowledge without relying on deep end-to-end model adaptation.
- question answeringQuestion answering determines an appropriate answer to a question expressed in natural language.
- overfittingOverfitting occurs when a model fits its training data too closely and consequently generalizes poorly to new data.
- slanted triangular learning rate scheduleA learning-rate schedule that linearly increases the learning rate and then linearly decreases it.
- shallow learningShallow machine learning uses methods with limited representation depth, typically with engineered features rather than learned deep feature hierarchies.
- low-resource languageA low-resource language is a language for which relatively little data or other computational resources are available.
- encoderAn encoder is a model component that transforms an input sequence into an internal representation.
- random forestA random forest is an ensemble learning method that generates multiple decision trees and combines their outputs.
- bag-of-words modelA bag-of-words model represents text by its words and their counts or weights while disregarding word order.
- logistic regressionA statistical classification method that models class probabilities using a logistic function of input features.
- Multilingual BERTmBERT is a multilingual version of BERT pretrained on text from multiple languages.
- carbon footprintThe greenhouse-gas emissions attributable to an activity, process, or product, commonly expressed as an equivalent amount of carbon dioxide.
- lr_findlr_find is a fastai utility method that helps identify a suitable learning rate by evaluating loss across learning rates.
- stop wordA word commonly filtered from text during processing because it is often considered less informative than other words.
- multi-head attentionMultihead attention is an attention mechanism that computes multiple representations of relationships within an input sequence.
- causal language modelCausal language modeling trains a model to predict the next token from preceding tokens without access to future tokens.
- bidirectional language modelA language model that combines forward and backward language models to use context from both directions in a sequence.
- attentionAttention is a neural-network technique that assigns focus to selected parts of an input when computing an output.
- implicit transfer learningImplicit transfer learning is the transfer of learned representations to another learning task without an explicitly separated transfer step.
- sequence-to-sequence modelA sequence-to-sequence model maps an input sequence to an output sequence, typically through an encoder and a decoder.
- tokenizerA tokenizer is a software component that divides text into tokens and may convert them into model-ready representations.
- support vector machineA supervised learning method that determines a separating boundary by maximizing its margin from the closest training points.
- supervised learningSupervised learning is a machine-learning paradigm in which a model learns from examples annotated with output labels.
- language modelingLanguage modeling is the task of modeling probability distributions over sequences of language.
- AWD-LSTMA fastai language-model architecture based on an LSTM with averaged stochastic gradient descent and randomly dropped weights.
- tokenizationThe process of dividing text into units such as words, subwords, or characters.
- Semantic Text Similarity BenchmarkThe Semantic Textual Similarity Benchmark is a dataset of sentence pairs with human similarity ratings.
- decoderA decoder is a model component that generates an output sequence from an internal representation.
- parameter reuseParameter sharing is the reuse of the same model parameters in multiple parts of a neural network.
- accuracyAccuracy is the proportion of evaluated samples whose predicted labels match their actual labels.
- DistilmBERTDistilmBERT is the multilingual distilled BERT model referred to in this section, corresponding to multilingual DistilBERT.
- sigmoidA sigmoid function maps real-valued inputs to values between zero and one, with an S-shaped curve.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
