Transfer learning is the reuse of knowledge or structure learned in one context to support performance in another. In language modeling, it can mean adapting a model trained on a large collection of text to a specific task or domain with much less additional data. section 3.5.6 section 9.5.2
The approach separates broad learning from specialization. A pretrained model captures patterns across its training data; those patterns can then support multiple applications rather than requiring a separate model to be built for each task. Word embeddings offer another example: vector spaces learned from one dataset can be adapted to a new domain. section 3.5.6 section 2.5.6
Specializing a language model
A broadly pretrained model can be further adapted using a smaller collection of legal or medical text. This reuses its prior language learning while specializing it for a domain. section 3.5.6
Transfer also matters when assessing whether a model can use learned structure beyond its original training objective. For example, mathematical reasoning can emerge from next-token prediction, and exposure to code can support structured reasoning beyond programming. section 9.5.2
From prediction to mathematics
A study tested models at different training stages on MathCAMPS, a synthetic dataset covering 44 mathematics skills. Models trained only on next-token prediction acquired skills in an order that correlated with a human-designed curriculum. This does not establish that they learn mathematics as children do, but it suggests that prediction can produce structures useful for mathematical reasoning. section 9.5.2
Reuse does not by itself establish human-like understanding. Embeddings capture patterns of word co-occurrence without grounding in real-world experience, and their distributed structure can be difficult to interpret. Their computational costs can also grow with model size and complexity. section 2.5.6















