generate() turns a prompt into generated text. It converts the prompt to token indexes, passes them to sample() to predict more indexes, then decodes the result. This lets a caller work with text while sample() handles next-token prediction. section 11.3.2
Listing 11.8 A function to generate text with GPT-2XL
def generate(prompt, max_new_tokens, temperature=1.0,
top_k=None):
if prompt == '':
x=torch.tensor([[tokenizer.encoder.encoder['<|endoftext|>']]],
dtype=torch.long) #1
else:
x = tokenizer(prompt) #2
y = sample(x, max_new_tokens, temperature, top_k) #3
out = tokenizer.decode(y.squeeze()) #4
print(out)
The GPT-2XL implementation shows the handoff: tokenize the prompt, call sample() with a token limit and sampling settings, then decode the sequence. An empty prompt starts from the end-of-text token. section 11.3.2
Choose the inputs to match the task:
- Set max_new_tokens to limit how many indexes sample() adds. Temperature and top_k control sampling; with no top-K limit and temperature 1, the earlier recurrent-model version matches its sample() baseline. section 11.3.2 section 8.5.2
- For GPT-2XL, use an empty prompt to start without preceding text, or supply a prompt for a continuation. In the trained GPT version, the prompt must be nonempty; "UNK" is used to start effectively unconditional generation. section 11.3.3 section 12.4.2 section 12.4.3
- For the trained GPT version, supply a saved weights file. Its sample() returns only new indexes so generate() can preserve the original prompt when it contains unknown words. section 12.4.2
Check a short continuation
Prompt GPT-2XL with “I went to the kitchen and” and request 10 new tokens. Repeating the call with different random seeds produces different continuations, including conversations and drinks, that fit the setting. Plausible wording is not a guarantee of factual accuracy. section 11.3.3
Generated passages can also resemble training data too closely. Compare outputs from saved checkpoints with the training text when that matters: in one test, passages from a 40-epoch model included a copied span. section 12.4.3
