Pretraining teaches a model from data before it specializes in a labeled task. It helps when the task has too few labels to teach every parameter well. An IMDb review classifier trained on 20,000 reviews began memorizing its training examples before it learned a good word embedding. A separate task using unlabeled reviews gives that embedding more opportunity to learn useful patterns. section 14.5.3 section 14.5.4
Pretrain an IMDb word embedding with CBOW
Continuous Bag of Words (CBOW) turns text into a prediction task: given four words on each side of a missing word, guess the word in the middle. For instance, a context containing “sail,” “wave,” and “mast” might suggest “boat.” Tokenize the reviews, slide a nine-token window across each sequence, and set aside its center token as the target. An Embedding layer maps the eight context tokens to vectors; averaging those vectors and predicting the target trains the embedding without positive or negative review labels. The example uses 64-dimensional vectors, sparse categorical crossentropy, and four training epochs. Its purpose is to learn embedding parameters that can be transferred to a classification model—not to use CBOW itself as the review classifier. section 14.5.4 section 14.6
Listing 14.25 Preprocessing our IMDb data for pretraining a CBOW model
context_size <- 4L #1
window_size <- context_size + 1L + context_size #2
window_data <- function(token_ids) {
windows <- tf$signal$frame(
token_ids,
frame_length = window_size,
frame_step = 1L
)
tensor_slices_dataset(windows)
}
split_label <- function(window) {
.[left, label, right] <-
tf$split(window, c(context_size, 1L, context_size))
bag <- tf$concat(tuple(left, right), axis = 0L)
tuple(bag, label)
}
dataset <- text_dataset_from_directory("aclImdb/train", batch_size = NULL) #3
dataset <- dataset |>
dataset_map(\(x, y) x, num_parallel_calls = 8) |> #4
dataset_map(tokenize_no_padding, num_parallel_calls = 8) |> #5
dataset_interleave(window_data) |> #6
dataset_map(split_label, num_parallel_calls = 8) #7Found 70000 files belonging to 3 classes. #7
Follow the preprocessing pipeline from raw reviews to CBOW training pairs: it discards review labels, tokenizes text, creates overlapping windows, and separates each center word from its surrounding context. section 14.5.4
The same principle scales beyond word embeddings. A language model can learn to predict the next token in a text sequence, then be fine-tuned for tasks such as text classification. Because its pretraining objective needs no curated input–answer pairs, it can use text at a much larger scale. That scale has a cost: pretraining a large Transformer demands substantial data, compute, and energy, so using an existing pretrained model is often more practical than training one anew. section 16.1 section 16.2.1 section 16.3
Reusing a pretrained model also means respecting how its inputs were prepared. For RoBERTa review classification, tokenize with its tokenizer and pack sequences with the start, end, and padding tokens used during pretraining. Matching that ordering helps fine-tuning train more quickly and accurately. section 15.4.3
