generate_text_simple is a simple autoregressive routine: given token IDs, it predicts one next token at a time, appends each prediction to the context, and repeats until max_new_tokens is reached. It uses greedy decoding, selecting the highest-scoring vocabulary entry at every step. section 4.7
Each iteration crops the input to the model's maximum context, runs the GPT model without gradient tracking, keeps the logits for the final sequence position, converts them to probabilities, selects the argmax token, and concatenates it along the sequence dimension. The returned tensor contains the original and generated token IDs. section 4.7
The text-generation pipeline connects several representations:
- Encode input text into token IDs and add a batch dimension. section 5.1.1
- Pass the token sequence through GPT to obtain logits for vocabulary entries at each position. section 5.1.1
Generating six tokens
Starting with "Hello, I am", encoding produces a tensor with shape [1, 4]. With max_new_tokens=6, the routine appends six predictions, producing a sequence of length 10. The generated IDs can then be decoded into text; an untrained model may produce incoherent output. section 5.1.1
