A foundational model is pretrained on a large text corpus without one specific task. Its learned representations or generation patterns can then support a narrower application, reducing the need to learn everything from task-specific data. section 7.4.2 section 10.2.2
A practical workflow treats the model as a starting point:
- Start with a pretrained model whose parameters were learned from another corpus or task. section 7.4.2
- Adapt it with fine-tuning data, such as labeled task examples or a focused text corpus. section 10.2.2
- Evaluate the adapted model rather than assuming that additional training improves generalization; training accuracy can continue rising while test accuracy declines. section 7.4.2 section 10.2.2
Fine-tuning GPT-2
A GPT-2 model can be initialized from pretrained weights, trained on a filtered collection of text lines, and evaluated on held-out lines. Hugging Face’s Trainer performs the fine-tuning; generation then shifts toward the supplied corpus, producing output distinct from vanilla GPT-2.
