Transfer learning

As explained in Inside Deep Learning: Math, Algorithms, Models

A training approach that adapts a pretrained model to a new task, usually by reusing learned weights and matching the original preprocessing expectations.

Transfer learning adapts a model trained on a large source dataset to a related target task. Reusing learned weights can reduce labeled-data, compute, and training-time requirements while improving accuracy. It works best when the source and target domains share useful structure, such as the local relationships common to images. section 13.0 section 13.1

A practical transfer-learning workflow is:

• Start with a pretrained architecture and replace its task-specific output layer. For a ResNet-18 classifier, use the existing input width: model.fc = nn.Linear(model.fc.in_features, 2). section 13.2
• Match preprocessing to the pretrained model. ImageNet-trained ResNet expects channel normalization with mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225]. section 13.2.2
• Choose between warm and frozen weights. Warm starting updates pretrained parameters during training; freezing sets requires_grad=False for the original layers and trains the replacement layer. section 13.2.3 section 13.2.4
• Use frozen weights when the labeled dataset is extremely small and warm weights when adapting more parameters is justified. Frozen models are generally more stable, while warm models can be more accurate. section 13.2.4 section 13.3

Adapting an ImageNet ResNet

model = torchvision.models.resnet18(pretrained=True) model.fc = nn.Linear(model.fc.in_features, 2) model = NormalizeInput(model) The convolutional filters retain features such as edges, colors, sharpening, and blurring, while the new fully connected layer learns the cat-versus-dog decision. Train the resulting model with the usual loss and optimizer. For frozen transfer, disable gradients before replacing fc. section 13.2.1 section 13.2.2 section 13.2.4

The same pattern applies to text. A pretrained transformer supplies broadly learned language representations, while a task-specific classifier is trained on the target labels. BERT-style models require their original tokenizer and encoding process; DistilBERT provides a smaller, faster transfer-learning option. section 13.4 section 13.4.1

Figure 13.1 Training a big model on a big dataset. It is a one-time cost, because we can reuse the big model by transferring it to many different smaller tasks. This is done by editing the model but keeping almost all of the original architecture and weights θ intact. Then we train the modified model on the new dataset.

Shows the one-time cost of training a large source model and the reuse pattern: modify a small part while preserving most architecture and weights for multiple target tasks. section 13.1

How Inside Deep Learning: Math, Algorithms, Models covers it

How deep it goes

Explains weight reuse, output-layer surgery, preprocessing, warm starts, frozen layers, dataset-size tradeoffs, and transformer adaptation.

Examples

Demonstrates cat-versus-dog ResNet classification, ImageNet filter visualization, tiny labeled datasets, AG News, BERT, and DistilBERT.

What it assumes

Assumes neural-network training and gradient descent, while teaching pretrained weights, output-layer replacement, and gradient freezing directly.

What you can do after

Apply it via PyTorch, torchvision, and Hugging Face by replacing output layers, matching tokenization or normalization, and selecting warm or frozen training.

How it connects

Connects pretrained models with convolutional neural networks, transformers, fine-tuning, warm starts, frozen weights, and preprocessing.

liveBookmanning.com

Read Inside Deep Learning: Math, Algorithms, Models on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  LLMs in ProductionTransfer learning is a machine-learning technique that adapts knowledge learned from one task or dataset to another task or domain.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionTransfer learning is reusing knowledge learned from one task or dataset to improve learning on another task or dataset.
• 

  liveBookmanning.com
  Generative AI in ActionTransfer learning adapts knowledge learned from one task or dataset to improve performance on another task.
• 

  liveBookmanning.com
  Knowledge Graphs and LLMs in ActionTransfer learning is a machine-learning technique that reuses patterns learned from one task or domain to improve performance on another task or domain.
• 

  liveBookmanning.com
  Time Series Forecasting Using Foundation ModelsTransfer learning is a technique that applies knowledge learned by a model on one dataset or task to a different dataset or task.
• 

  liveBookmanning.com
  Grokking Machine LearningTransfer learning is a technique that adapts a pretrained model to a new task or dataset by continuing its training, often by modifying only some parameters.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA technique that adapts knowledge learned from one task or dataset to a related task.
• 

  liveBookmanning.com
  Sutskever's ListTransfer learning is a technique that adapts knowledge learned from one task or dataset to another task or dataset.
• 

  liveBookmanning.com
  Domain-Specific Small Language ModelsThe process of taking a model trained on one task or domain and applying it to a related task or domain, often with further tuning.
• 

  liveBookmanning.com
  Outlier Detection in PythonTransfer learning is a technique that adapts knowledge learned by a model on one task or dataset to another task or dataset.

Compare how 11 books explain transfer learning →

Related concepts

• fine-tuningused by · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
• pretrained modeluses · A pretrained model is a machine-learning model whose parameters were learned through prior training before it is used or adapted for a subsequent task.
• embeddinguses · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
• generalizationuses · Generalization is the extension of a pattern, model, solution, or conclusion beyond the specific examples or cases from which it was derived.
• unlabeled datauses · Unlabeled data consists of examples that lack target labels or task-specific annotations.
• causal invarianceused by · Causal invariance is the stability of a causal relationship across changes in environments or data distributions.

This explainer was generated by Manning's knowledge graph from the text of Inside Deep Learning: Math, Algorithms, Models. The section links point to the parts of the book it draws on.