Predicting an entire sentence as one output is impractical: a 20,000-word vocabulary allows 160 quadrillion four-word sequences. A language model makes the problem manageable by predicting one token at a time. Given past tokens, it estimates a probability distribution over what comes next; repeating that prediction generates text. section 15.1
Generate Shakespeare-like text
A character-level model starts with a prompt such as “KING RICHARD III:”. Feed the prompt’s characters through the model to establish its starting state, choose a predicted next character, then feed that character back in. Repeat to produce dialogue-like text. The resulting passage imitates the style without being a quotation. section 15.1.2
Listing 15.8 Predicting with the language model a token at a time
import numpy as np generated_ids = [] max_length = 250 for i in range(max_length): #1 next_char = int(np.argmax(predictions, axis=-1)[0]) #2 generated_ids.append(next_char) inputs = keras.ops.expand_dims([next_char], axis=0) predictions, state = generation_model.predict((inputs, state), verbose=0)
The generation loop shows the step that a single prediction call cannot perform: each selected token becomes the next input. section 15.1.2
Related setups change what information is available when predicting tokens:
- A masked language model predicts tokens from surrounding tokens and can support text or token classification. section 15.6
- A sequence-to-sequence language model predicts target tokens using both earlier target tokens and a separate source sequence; translation is one application. section 15.6
- A Transformer decoder uses causal attention for next-token prediction, while a Transformer encoder uses bidirectional attention to represent a sequence. section 15.6
Generating fluent text is not the same as reliably solving a problem. A model may answer a familiar math question correctly yet fail when names or numbers change. Prompts that ask it to show intermediate steps can improve results, but individual outputs still need checking. section 16.4.4
For large language models, bigger is not the only practical choice. With a fixed pretraining compute budget, training a smaller model on more data may outperform training a larger one; deployment cost also favors models that fit cheaper hardware. section 16.5
