A foundation model is trained on broad data so it can support many downstream tasks instead of being built for just one. Its training is generally self-supervised: the data supplies its own target, such as the next word in a sentence or a sharp image to reconstruct from a noisy one. That lets training use large amounts of unlabeled data. section 16.4.2 section 1.8
Turn a pretrained model into a review classifier
Load a pretrained RoBERTa backbone and its tokenizer with KerasHub. The backbone converts an input sequence into contextual representations; it does not yet produce a review label. Take the first token’s representation, add a classification head, then fine-tune on IMDb reviews. The resulting classifier reached 93% test accuracy after one epoch. Reusing a large model improved on the earlier 90% ceiling, though it was more expensive to run. section 15.4.2 section 15.4.4
Listing 15.22 Extending the base RoBERTa model for classification
inputs <- backbone$input outputs <- inputs |> backbone() |> op_subset(, 1, ) |> #1 layer_dropout(0.1) |> layer_dense(768, activation = "relu") |> layer_dropout(0.1) |> layer_dense(1, activation = "sigmoid") classifier <- keras_model(inputs, outputs)
The R/Keras listing shows the task-specific head attached to RoBERTa: select one sequence representation, then produce a binary prediction. section 15.4.4
Choose how much of a pretrained model to adapt:
- Prompt it when its existing representations can answer the request without task-specific retraining. section 1.8
- Use feature extraction: keep a pretrained backbone and train a new head on its outputs. For image classification, the convolutional base supplies reusable visual features while the original classifier is tied to its training classes. section 8.3.1
- Fine-tune when the representations need adjustment. First train the new head with the base frozen; then unfreeze the base and train them jointly. Unfreezing before the head is trained can destroy useful pretrained representations. section 8.3.2
A backbone is the reusable, not-yet-task-specialized part of a model—not a synonym for every foundation model. Its reuse extends beyond classification: a pretrained ResNet backbone can receive a new head for object detection. Foundation models can also span modalities; a multimodal language model can incorporate image inputs alongside text. section 15.4.2 section 12.2.2 section 16.4.2
