Generation produces a new sequence from a starting input. For text, a model can predict the next token, add that prediction to its input, and repeat. This one-at-a-time process is called autoregressive generation: each new output becomes context for the next prediction. section 9.4
Generating a name
Start a character-level model with the marker $. It predicts a character; append that character and ask for another. Continue until an ending $ is produced. Training pairs each available prefix with its next character, so the model learns to use what precedes a character when making its prediction. section 9.4
# In[]:
def generate_samples(model, num_samples=1, max_len=max_name_length):
sequences = torch.zeros((num_samples, 1)).int()
for _ in range(max_len):
logits = model(sequences)
logits = logits[:, -1, :]
probs = F.softmax(logits, dim=-1)
idx_next = torch.multinomial(probs, num_samples=1)
sequences = torch.cat((sequences, idx_next), dim=1)
for sequence in sequences:
indices = torch.where(sequence == 0)[0]
end = indices[1] if len(indices) > 1 else max_len
sequence = sequence[1:end]
print(decode(sequence))
generate_samples(model, num_samples=10)
# Out[]:
jatini
tladauh
ceney
aamsse
keisvta
yassi
aenmli
lydude
ikiyo
jayhal
This PyTorch sampling function takes the last position’s logits, converts them to probabilities with softmax, samples a character with torch.multinomial, and appends it. It runs for a maximum number of steps, then trims the printed result at an end marker if one appears. section 9.4
The same generation loop can use different next-token models. A GPT-style transformer decoder uses attention to produce output sequences. Causal masking prevents its attention from looking at future positions while predicting from earlier ones. section 9.6.1 section 9.5.2
