Sampling chooses the next token randomly according to a model’s predicted probabilities. A likely token is more likely to be chosen, but it is not guaranteed. Unlike always taking the highest-probability token, sampling can produce different continuations from the same prompt. section 8.5.1 section 8.5.2
For text generation with an LSTM, start with a prompt and set a target length. Feed the current token sequence to the model, use softmax to turn its final-position logits into probabilities, sample one vocabulary index with NumPy’s `random.choice`, and append the corresponding word. Repeat until the target length is reached. If the sequence exceeds 100 tokens, use only its last 100 as input. section 8.5.1
Two controls change which tokens are likely to be sampled:
- Temperature divides logits before softmax. Below 1 favors likely tokens and reduces variation; above 1 increases variation but can make text less coherent. A temperature of 1 leaves logits unchanged. section 8.5.2
- Top-K restricts selection to the K most probable tokens and samples from their renormalized probabilities. A small K narrows the choices; no top-K restriction allows the full vocabulary. section 8.5.2
One prompt, different continuations
Repeatedly generating one word after “I'm not going to see” produces choices such as “you,” “those,” and “her” at temperature 1 without top-K restriction. With top-K set to 3 and temperature 0.5, the shown results concentrate on “you,” “the,” and “her.” At temperature 2 without top-K restriction, less expected choices such as “scarlatina” appear. section 8.5.2
The same loop works with token indexes in a GPT model: sample an index, append it, and predict again, stopping at a token limit or the end-of-text token. For music, event indexes replace word indexes; a short MIDI-derived prompt can be extended into a longer sequence and decoded back to MIDI. That music implementation uses temperature but omits top-K. section 11.3.2 section 14.4.2
Listing 14.7 A sample() function in music generation
softmax=torch.nn.Softmax(dim=-1)
def sample(prompt,seq_length=1000,temperature=1):
gen_seq=torch.full((1,seq_length),389,dtype=torch.long).to(device)
idx=len(prompt)
gen_seq[..., :idx]=prompt.type(torch.long).to(device)
while(idx < seq_length): #1
y=softmax(model(gen_seq[..., :idx])/temperature)[...,:388] #2
probs=y[:, idx-1, :]
distrib=torch.distributions.categorical.Categorical(probs=probs)
next_token=distrib.sample() #3
gen_seq[:, idx]=next_token
idx+=1
return gen_seq[:, :idx] #4
The music `sample()` listing shows the reusable loop: calculate next-event probabilities, draw with a categorical distribution, append the event, and continue to the requested sequence length. section 14.4.2
