Temperature controls how varied the next token can be during generation. Divide the model’s logits by the temperature before applying softmax and sampling from the resulting probabilities. At the default value of 1, the logits are unchanged. section 8.5.2
Choose a value for the kind of output you need:
- Below 1, such as 0.8, favors more likely tokens. Output tends to be conservative and can become repetitive. section 8.5.2 section 8.6
- Above 1, such as 1.5, gives less likely tokens more chances. Output becomes more varied, but may lose coherence or relevance. section 8.5.2
- Top-K sampling is a separate control: it limits selection to the K most likely tokens. You can use it alongside temperature. section 8.5.2
Compare next-word choices
With the prompt “I’m not going to see,” ten samples at temperature 1 include continuations such as “you,” “her,” and “my.” At temperature 2, with top-K disabled in both runs, samples also include less expected words such as “scarlatina” and “misery.” Diversity increases, but a surprising word is not necessarily a useful one. section 8.5.2
Listing 8.6 Generating text with temperature and top-K sampling
def generate(model, prompt , top_k=None,
length=200, temperature=1):
model.eval()
text = prompt.lower().split(' ')
hc = model.init_hidden(1)
length = length - len(text)
for i in range(0, length):
if len(text)<= seq_len:
x = torch.tensor([[word_to_int[w] for w in text]])
else:
x = torch.tensor([[word_to_int[w] for w in text[-seq_len:]]])
inputs = x.to(device)
output, hc = model(inputs, hc)
logits = output[0][-1]
logits = logits/temperature #1
p = nn.functional.softmax(logits, dim=0).detach().cpu()
if top_k is None:
idx = np.random.choice(len(logits), p=p.numpy())
else:
ps, tops = p.topk(top_k) #2
ps=ps/ps.sum()
idx = np.random.choice(tops, p=ps.numpy()) #3
text.append(int_to_word[idx])
text=" ".join(text)
for m in ",.:;?!$()/_&%*@'`":
text=text.replace(f" {m}", f"{m} ")
text=text.replace('" ', '"')
text=text.replace("' ", "'")
text=text.replace('" ', '"')
text=text.replace("' ", "'")
return text
In generate(), inspect `logits = logits/temperature` immediately before softmax. The optional `top_k` argument then provides a separate way to restrict the sampled words. section 8.5.2
The same control works for musical-event tokens. A music Transformer can start from a short MIDI-derived prompt, repeatedly sample the next event, and decode the completed sequence to MIDI. Its sampling function exposes temperature but omits top-K for simplicity. section 14.4.2
Listing 14.7 A sample() function in music generation
softmax=torch.nn.Softmax(dim=-1)
def sample(prompt,seq_length=1000,temperature=1):
gen_seq=torch.full((1,seq_length),389,dtype=torch.long).to(device)
idx=len(prompt)
gen_seq[..., :idx]=prompt.type(torch.long).to(device)
while(idx < seq_length): #1
y=softmax(model(gen_seq[..., :idx])/temperature)[...,:388] #2
probs=y[:, idx-1, :]
distrib=torch.distributions.categorical.Categorical(probs=probs)
next_token=distrib.sample() #3
gen_seq[:, idx]=next_token
idx+=1
return gen_seq[:, :idx] #4
The music sampler divides model output by `temperature`, applies softmax, samples an event with `Categorical`, and appends it to the sequence. section 14.4.2
