A language model can predict the next token, but a single prediction is not yet a passage of text. Generation turns that prediction into a sequence: choose a token, add it to the input, predict again, and repeat until the desired length is reached. This feedback loop is called autoregressive generation. section 15.1.2
Continue a prompt one character at a time
Start with a prompt such as “KING RICHARD III:”. Convert its characters to token IDs and feed them through a trained GRU one at a time to establish its starting state. Then select the most likely predicted character, feed it back alongside the updated state, and repeat for 250 characters. Convert the resulting IDs back to characters to read the continuation. section 15.1.2
The GRU example needs a small inference-time change. Training used fixed sequences of 100 tokens and handled state implicitly; generation accepts one token and an explicit state, then returns both the next-token prediction and the new state. The inference model can reuse the trained model’s weights because their computational structures match. Keep passing the state forward: it carries information encoded from earlier characters, including the prompt. section 15.1.2
Listing 15.8 Predicting with the language model a token at a time
import numpy as np generated_ids = [] max_length = 250 for i in range(max_length): #1 next_char = int(np.argmax(predictions, axis=-1)[0]) #2 generated_ids.append(next_char) inputs = keras.ops.expand_dims([next_char], axis=0) predictions, state = generation_model.predict((inputs, state), verbose=0)
The loop selects a character with argmax, appends its ID, and passes it back to the model with the updated state. section 15.1.2
Choosing a token is a separate decision from predicting it. The model produces scores across its vocabulary, not just one answer. Always taking the highest-scoring token is greedy search. It is simple, but it can settle into repeated phrases; repetition can follow from the next-token training objective rather than a broken loop. section 16.2.4
To change the character of a continuation, change the sampling strategy:
- Greedy search takes the highest-scoring token at every step. section 16.2.4
- Random sampling draws from the model’s distribution, exploring more possible continuations. Temperature changes that distribution before sampling; the examples range from highly erratic text at 2.0 to repetitive text at lower settings. section 16.2.4
- Top-K sampling limits the draw to K candidate tokens. The worked examples compare K values of 5 and 20 and also combine top-K with temperature. section 16.2.4
In practice, first make the prompt-to-token-to-text loop work, then inspect its output before tuning sampling. More variation can help escape repeated high-probability phrases, but the shown samples make clear that variation alone does not guarantee coherent text. section 15.1.2 section 16.2.4
