A text-generation `generate()` function turns a prompt into token indexes, asks a model to extend the sequence, and converts the indexes back into readable text. It wraps the lower-level `sample()` function, which predicts and selects subsequent indexes. In the GPT-2XL version, `generate()` handles tokenization and decoding while `sample()` performs the repeated predictions. section 11.3.2
A GPT-2XL call follows four steps:
- Tokenize the prompt; an empty prompt starts with the `<|endoftext|>` index. section 11.3.2
- Call `sample()` to predict one index at a time. It uses the most recent indexes when the sequence exceeds the model’s block size and stops after `max_new_tokens` predictions or an `<|endoftext|>` prediction. section 11.3.2
- Use `temperature` and `top_k` to change the sampling distribution. Their defaults, 1.0 and `None`, leave top-K filtering off. section 11.3.2 section 8.5.2
- Decode the resulting index sequence and print the text. section 11.3.2
Listing 11.8 A function to generate text with GPT-2XL
def generate(prompt, max_new_tokens, temperature=1.0,
top_k=None):
if prompt == '':
x=torch.tensor([[tokenizer.encoder.encoder['<|endoftext|>']]],
dtype=torch.long) #1
else:
x = tokenizer(prompt) #2
y = sample(x, max_new_tokens, temperature, top_k) #3
out = tokenizer.decode(y.squeeze()) #4
print(out)
The GPT-2XL listing shows the wrapper directly: initialize indexes, pass them to `sample()` with the generation settings, then decode the result. section 11.3.2
Test one continuation and then a passage
To see sampling choices clearly, append just one token to “I ’ m not going to see” across repeated calls. In the LSTM example, `top_k=3` and `temperature=0.5` produce more repeated choices than the defaults; `temperature=2` produces less predictable ones. For a longer GPT-2XL test, add ten tokens to “I went to the kitchen and” with several random seeds. The continuations tend to fit the setting, but fluent generated text need not be factually accurate. section 8.5.2 section 11.3.3
The trained-GPT version adds a `weights` argument to select saved model weights. It rejects an empty prompt and preserves the original prompt in the returned text when its vocabulary maps unknown words to `UNK`; `UNK` itself can serve as a starting prompt for generation tests. Its displayed implementation accepts `temperature` and `top_k` but calls `sample()` with fixed defaults, so those arguments would need forwarding before they could affect that version’s output. A separate `DDIMScheduler.generate()` method generates images by denoising random noise; it is not this prompt-to-text function. section 12.4.2 section 12.4.3 section 15.4.2
