Temperature sampling prepares next-token probabilities by rescaling model logits and applying softmax. The workflow uses a ten-entry toy vocabulary before continuing to top-p filtering. section 4.5.1
Toy temperature-sampling pipeline
Given ten toy logits, scale them with temperature 1.0, apply softmax, and plot the resulting token probabilities before filtering. section 4.5.1
Listing 4.9 Temperature scaling and sampling with toy data
toy_logits = torch.tensor( #1
[-0.7, -3.0, 0.1, -1.2, 2.0, -1.0, -0.5, -2.0, 0.3, 1.5]
)
toy_logits_scaled = scale_logits_by_temperature(toy_logits, 1.0) #2
toy_probas = torch.softmax(toy_logits_scaled, dim=-1) #3
plt.bar( #4
torch.arange(len(toy_probas)), toy_probas,
alpha=0.5
)
plt.ylim([0, 1])
plt.xlabel("Vocabulary index")
plt.ylabel("Probability")
plt.show()
Inspect the PyTorch listing that scales toy logits by temperature and converts them into token probabilities. section 4.5.1
