liveBook · Manning

Temperature

As explained in Build an Advanced RAG Application (From Scratch)

Temperature is a decoding parameter that controls the sharpness of a language model's next-token probability distribution.

Temperature is a decoding parameter that reshapes the next-token probability distribution. Sampling remains probabilistic, while temperature controls whether generation favors predictable or surprising continuations. section 5.2.3

Adjusting temperature changes the balance between determinism and diversity:

  • T > 1 flattens the distribution, increasing randomness and creative variation while raising the risk of irrelevant or nonsensical text. section 5.2.3
  • T < 1 sharpens the distribution, favoring highly probable words and producing more deterministic output. section 5.2.3
  • A temperature close to zero approaches greedy decoding, which selects the most probable continuation rather than exploring alternatives. section 5.2.3

Same prompt, different behavior

For the same next-token distribution, a low temperature repeatedly favors the leading continuation, whereas a high temperature gives less probable alternatives more opportunity to be selected. This makes temperature useful for tuning predictable versus creative generation. section 5.2.3

Formula: P(w_i) = exp(log(P(w_i)) / T) / sum_j(exp(log(P(w_j)) / T))

The distribution is computed by dividing each token's log-probability by T and normalizing the exponentiated values. section 5.2.3

How Build an Advanced RAG Application (From Scratch) covers it

How deep it goes
Defines log-probability scaling and directional effects, but stops short of implementation code or parameter-selection procedures.
Examples
Contrasts high-temperature creative output with low-temperature deterministic output and near-zero behavior approaching greedy decoding.
What it assumes
Assumes familiarity with next-token probabilities and distinguishes probabilistic sampling from deterministic greedy decoding.
What you can do after
Apply it via decoder settings to trade predictable generation against creative diversity and output risk.
How it connects
Connects sampling with greedy decoding, top-k sampling, top-p sampling, presence penalties, and retrieval-augmented generation.

Also explained in

Compare how 20 books explain temperature →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build an Advanced RAG Application (From Scratch). The section links point to the parts of the book it draws on.