liveBook · Manning

Transfer learning

As explained in Build an Advanced RAG Application (From Scratch)

Transfer learning is a machine-learning technique that adapts knowledge learned for one task or domain as the starting point for another.

Transfer learning adapts a model pre-trained on one task as the starting point for a related but different task. It can improve performance while reducing training time. section 2.1

Within Transformer-based NLP systems, pre-training on vast amounts of unlabeled text supplies reusable knowledge before task-specific training. This complements parallel processing and self-attention, which capture long-range relationships. section 2.1

Transformer pre-training

A model pre-trained on one language task becomes the starting point for a different, related task instead of beginning training from scratch. section 2.1

Figure 2.1 The strength of the transformer model is in its ability to understand the significance and context of every word in a sentence, unlike RNNs which primarily assess the connection of a word with its immediate neighbor.

Use the figure to connect Transformer self-attention with its ability to capture word significance, context, and long-range relationships. section 2.1

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Defines pre-training transfer and its benefits, but stops short of implementation steps, fine-tuning details, or quantitative trade-offs.
Examples
Uses Transformer pre-training on vast amounts of unlabeled text as the transfer-learning scenario.
What you can do after
Evaluate transfer learning as a way to improve performance and reduce training time when adapting a pre-trained model.
How it connects
Connects transfer learning with pre-training, Transformers, self-attention, natural language processing, training, and inference.

Also explained in

Compare how 16 books explain transfer learning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.