A pretrained model is a machine-learning model whose parameters were learned from earlier training data. A sufficiently large and general training set can give the model reusable features, allowing it to be adapted to a new problem with little data and relatively few changes. section 8.3 section 8.2.1
Use a pretrained model in two main ways:
- Feature extraction reuses the learned representations while training a new task-specific classifier. section 8.3
- Fine-tuning adapts some or all of the pretrained model to the new task. section 8.3
- Transfer works across related or even substantially different problems when the original training data is broad enough. section 8.3
From ImageNet to new tasks
An Xception convolutional neural network trained on ImageNet’s 1.4 million labeled images and 1,000 classes can bootstrap dogs-versus-cats classification or help identify furniture, despite those target classes differing from the original task. section 8.3
The same pattern extends beyond classification. SAM supplies general segmentation knowledge from 11 million images and more than 1 billion object instances without requiring fine-tuning first. Pretrained Transformers support text classification, while pretrained generative models make large-scale language generation practical without repeating prohibitively expensive pretraining. section 11.4 section 15.4 section 16.3
compiled_generate <- function(prompt, sample_fn, max_length = 64) {
tokens <- as.array(tokenizer(prompt))
prompt_length <- length(tokens)
tokens[seq(prompt_length + 1, max_length)] <- 0L
dim(tokens) <- c(1, max_length)
storage.mode(tokens) <- "integer"
for (i in seq(prompt_length, max_length - 1)) {
prediction <- predict(mini_gpt, tokens, verbose = 0)
prediction <- prediction[, i, ]
next_token <- sample_fn(prediction) - 1L
tokens[, i + 1] <- as.array(next_token)
}
tokenizer$detokenize(tokens)
}
Inspect a compact generation loop that repeatedly predicts the next token with a pretrained or language model and delegates token choice to a sampling strategy. section 16.2.4
