liveBook · Manning

Temperature scaling

As explained in Build an Advanced RAG Application (From Scratch)

Temperature sampling modifies next-token probabilities by scaling their log-probabilities with a temperature value before sampling.

Temperature scaling adjusts the next-token probability distribution before sampling by dividing log-probabilities by a temperature value. It controls the balance between deterministic, repetitive text and more creative, random output. section 5.2.3

Use the temperature value to change sampling behavior:

  • Set T > 1 to increase randomness and encourage more surprising, creative outputs, while accepting a greater risk of irrelevant or nonsensical text. section 5.2.3
  • Set T < 1 to make generation more deterministic by favoring the most probable tokens. section 5.2.3
  • Move T close to 0 to approach greedy decoding. section 5.2.3
  • Treat the temperature as a smooth randomness control, but tune it carefully because high values can produce gibberish and low values can limit creativity. section 5.2.3

Low versus high temperature

For a prompt such as "The quick brown fox," a low temperature favors the most probable continuation and produces more repeatable text. A high temperature explores less-probable continuations, increasing diversity while potentially reducing coherence. section 5.2.3

Formula: P(w_i) = exp(log(P(w_i)) / T) / sum_j(exp(log(P(w_j)) / T))

The probability transformation divides each token's log-probability by T, exponentiates the result, and normalizes across candidate tokens before sampling. section 5.2.3

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Explains log-probability scaling, temperature effects, and trade-offs without detailing a full decoder implementation.
Examples
Contrasts near-zero, low, and high temperatures through deterministic, repetitive, creative, and potentially nonsensical outputs.
What it assumes
Uses next-token probabilities and probabilistic sampling as its immediate foundation.
What you can do after
Apply temperature scaling to tune generated text toward repeatability or creativity.
How it connects
Connects sampling methods with autoregressive language models, greedy decoding, top-k sampling, and top-p sampling.

Also explained in

Compare how 2 books explain temperature scaling →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.