A sequence model may need more text to learn useful representations than a labeled task can provide. Pretraining tackles that mismatch: first train parameters on a task that uses unlabeled text, then reuse them for the task you care about. The text can come from the target domain or from broader sources in the relevant language. section 14.5.4 section 14.6
Prime an embedding for review classification
Suppose an LSTM classifies IMDb reviews using learned word vectors. With about 20,000 labeled training reviews, it can start memorizing reviews before its embedding becomes as useful as it could be. Instead of asking sentiment labels to teach the embedding everything, add 25,000 unlabeled reviews and train the embedding on a separate task. A Continuous Bag of Words (CBOW) model slides a window through each review and predicts the middle word from its neighbors: given words such as “sail,” “wave,” and “mast,” it might predict “boat.” The resulting word vectors can be transferred to the classifier. section 14.5.3 section 14.5.4 section 14.6
For the CBOW version, the practical steps are:
- Keep the classifier’s vocabulary, but tokenize the pretraining text without truncating or padding it. section 14.5.4
- Slide a nine-token window across the text; use four tokens on each side as input and the center token as the label. section 14.5.4
- Train an Embedding layer in the CBOW model, then reuse its learned vectors for the downstream sequence model. section 14.5.4 section 14.6
Listing 14.27 Preprocessing our IMDb data for pretraining a CBOW model
import tensorflow as tf context_size = 4 #1 window_size = 9 #2 def window_data(token_ids): num_windows = tf.maximum(tf.size(token_ids) - context_size * 2, 0) windows = tf.range(window_size)[None, :] windows = windows + tf.range(num_windows)[:, None] windowed_tokens = tf.gather(token_ids, windows) return tf.data.Dataset.from_tensor_slices(windowed_tokens) def split_label(window): left = window[:context_size] right = window[context_size + 1 :] bag = tf.concat((left, right), axis=0) label = window[4] return bag, label dataset = keras.utils.text_dataset_from_directory( #3 imdb_extract_dir / "train", batch_size=None #3 ) #3 dataset = dataset.map(lambda x, y: x, num_parallel_calls=8) #4 dataset = dataset.map(tokenize_no_padding, num_parallel_calls=8) #5 dataset = dataset.interleave(window_data, cycle_length=8, num_parallel_calls=8)#6 dataset = dataset.map(split_label, num_parallel_calls=8) #7
The preprocessing listing makes the unlabeled training task concrete: each sliding window becomes surrounding-word inputs paired with its center word. section 14.5.4
Pretraining also applies beyond word embeddings. A mini-GPT example tokenizes a portion of the C4 text corpus and forms input–target pairs by shifting each token sequence one position: the model learns to predict the next token. This needs much more data and compute than the small CBOW exercise. For large Transformers, using an existing pretrained model is often more practical than repeating pretraining; a roughly one-billion-parameter Gemma model, for example, was pretrained on about two trillion tokens. section 16.2 section 16.3
Rule of thumb: pretrain when unlabeled text is plentiful but task labels are scarce; transfer what the unlabeled task learned rather than expecting a small labeled set to learn every representation. At LLM scale, weigh data, model size, compute, and deployment cost together. More parameters alone are not the goal, and high-quality public pretraining text is becoming harder to find. Pretraining is an initial phase, not a substitute for later task-specific or instruction tuning. section 14.5.4 section 16.5 section 16.4.1
