Temperature controls how concentrated a model’s next-token probability distribution is during sampling. Lower values favor the most likely tokens; higher values spread probability across more alternatives. section 16.2.4
Changing temperature
With temperature 0.2, generation begins to resemble greedy search and repeats patterns. At 0.8, it produces more coherent variation. At 2, it becomes highly diverse and may produce seemingly random text. section 16.2.4
Implementation divides the model’s logits by temperature before reshaping them and sampling with random_categorical(). The resulting value changes exploration without removing any token from consideration. section 16.2.4
Temperature differs from top-k sampling: temperature changes relative likelihoods, whereas top-k sets probabilities outside the selected candidates to zero. Both strategies can be combined. section 16.2.4
random_sample <- function(preds, temperature = 1) {
preds <- preds / temperature
preds <- op_reshape(preds, c(1, -1))
random_categorical(preds, num_samples = 1) |> op_squeeze()
}cat(compiled_generate(prompt, random_sample))
A piece of advice, just read my knees and stick with getables, and a hello to
me. However, the bar napkin doesn’t last as long. I happen to be waking up close
and pull it up as I wanted too and I still get it, really, shouldn’t be a
reaction)
Inspect a complete random-sampling function and compare generated text at the default temperature. section 16.2.4
compiled_generate(prompt, \(preds) top_k(preds, k = 5, temperature = 0.5)) A piece of advice that you can use to get rid of the problem. The first thing you need to do is to get the job done. It is important that you have a plan that will help you get rid of it. The first thing you need to do is to get rid of the problem yourself.
Compare combined top-k and temperature sampling, including a lower-temperature output. section 16.2.4
