Generation produces new text by repeatedly predicting a next token and adding it to the sequence. Unlike a request to answer a question or write about a topic, unconditional generation supplies no task-specific instruction. A trained GPT-style model can use the generic token `UNK` as its starting prompt. section 12.4.2 section 12.4.3
The model still needs a starting token: the `generate()` function rejects an empty prompt. Here, `UNK` is a practical stand-in for an empty, task-free prompt—not a claim that the model runs with no input at all. Contrast it with a specific opening phrase, which steers the passage toward that phrase. section 12.4.2 section 12.4.3
To generate a passage with a trained GPT-style model:
- Convert the starting token to its integer index and feed it to the model. For the unconditional test, use `UNK`. section 12.4.2 section 12.4.3
- Predict a next-token index, sample from the resulting probabilities, append the index, and repeat until the requested number of new tokens is reached. The sampler can limit its input to the model’s block size. section 12.4.2
- Convert the new indexes back to text. Temperature and top-K are available as sampling controls; the saved weights select which trained model to test. section 12.4.2
Listing 12.7 A sample() function to predict subsequent indexes
def sample(idx, weights, max_new_tokens, temperature=1.0, top_k=None):
model.eval()
model.load_state_dict(torch.load(weights,
map_location=device)) #1
original_length=len(idx[0])
for _ in range(max_new_tokens): #2
if idx.size(1) <= config.block_size:
idx_cond = idx
else:
idx_cond = idx[:, -config.block_size:]
logits = model(idx_cond.to(device)) #3
logits = logits[:, -1, :] / temperature
if top_k is not None:
v, _ = torch.topk(logits, top_k)
logits[logits < v[:, [-1]]] = -float('Inf')
probs = F.softmax(logits, dim=-1)
idx_next=torch.multinomial(probs,num_samples=1)
idx = torch.cat((idx, idx_next.cpu()), dim=1) #4
return idx[:, original_length:] #5
The sampling loop shows how a decoder-only model extends token indexes one at a time and where temperature and top-K act on its next-token scores. section 12.4.2
Test for copied passages
Set `prompt="UNK"`, generate 20 new tokens repeatedly, and fix random seeds so runs can be compared. Passages from a model trained for 20 epochs were grammatical and Hemingway-like without directly copying the training text. With 40 epochs, part of one passage was copied. Check output against the training file rather than treating fluent prose alone as success. section 12.4.3 section 12.5
Training text shapes the generated style. For the small GPT-style model, three Hemingway novels provide variety without making the token set too difficult to train in the allotted time. Generic-prompt generation is therefore useful both for producing stylistic samples and for checking whether a model has memorized passages. section 12.1.2 section 12.4.3
