Training a large model from scratch is often unnecessary. A foundational, or foundation, model learns broadly useful representations from large-scale data, then serves many downstream tasks with little retraining or none. Such models are generally trained with self-supervision: for example, predicting missing or subsequent content from the input itself. A pretrained model is not automatically a foundation model; an ImageNet classifier can be useful for reuse without having been trained on broad data with a self-supervised objective. section 16.4.2 section 8.3
Reuse an image model for a smaller dataset
Suppose you have a small dogs-versus-cats dataset. Start with an Xception convolutional backbone pretrained on ImageNet, rather than learning every visual feature anew. Run images through its matching preprocessor and backbone, then train a new classifier on the resulting features. Reuse the backbone, not its original classifier: the latter predicts ImageNet classes, while convolutional feature maps retain more general visual information. section 8.3 section 8.3.1
Figure 8.12 Swapping classifiers while keeping the same convolutional base
The diagram shows the key swap: keep a learned convolutional base and replace the task-specific classifier. section 8.3.1
Choose the amount of adaptation the new task needs:
- Use the existing model directly when its capabilities fit the task. Prompting can elicit responses from generative models without retraining; SAM can segment new objects when given an example. section 1.8 section 11.3
- Extract features and train a new head when the representations transfer but the outputs must change. For image tasks, earlier convolutional layers learn more generic features; later layers are more specialized. section 8.3.1
- Fine-tune when the reused representations need adjustment. Add a head, freeze the base, train the head, then unfreeze appropriate base layers and train jointly. Do not start by training a random head against an unfrozen base: its large error signal can damage learned representations. section 8.3.2
Reuse also works beyond image classification. A pretrained ResNet backbone can supply features for an object detector, while a pretrained RoBERTa Transformer can gain a classification head for movie-review sentiment. For text, the tokenizer, architecture, and weights must match pretraining; differently assigned token IDs would undermine the learned representations. Loading a preset avoids rebuilding that combination by hand. section 12.2.2 section 15.4.2 section 15.4.4
Listing 15.20 Loading the RoBERTa pretrained model with KerasHub
import keras_hub tokenizer = keras_hub.models.Tokenizer.from_preset("roberta_base_en") backbone = keras_hub.models.Backbone.from_preset("roberta_base_en")
These two preset calls load the matching RoBERTa tokenizer and backbone for a downstream text task. section 15.4.2
Rule of thumb: reuse expensive pretraining, but budget for inference and adaptation. Large pretrained Transformers can outperform simpler classifiers while costing more to run. Pretraining a generative model can also demand substantial compute and electricity, which makes an existing model the practical starting point for many applications. section 15.4.4 section 16.3
