A language model generates text one token at a time. At each step, it ranks possible next tokens using the text generated so far. Temperature is a decoding setting that affects how often generation moves beyond the most probable choice. That matters when a chatbot needs varied replies rather than the same continuation every time. section 10.2 section 12.5.2
Figure 10.2 Stochastic chameleons deciding words one at a time
Follow the token choices along the diagram: a generated sentence can take a lower-ranked branch instead of selecting the most probable token at every step. section 10.2
Greedy search always takes the most probable next token. It can get stuck repeating itself. Increasing temperature—typically above 1.0—makes selection less greedy and more creative; a repetition penalty is another way to address loops. Temperature changes the choice process, not the model’s ability to check whether an answer is correct. section 10.2.1 section 12.5.2
Configure a chatbot
A LangChain Replicate configuration passes `input={"temperature": 0.5, "max_length": 100, "top_p": 1}` to a chat model. Run the same prompt several times and compare the replies. A nonzero temperature can produce different responses, so score candidates against the conversation’s goals rather than assuming that variety means quality. section 12.5.2
For a math tutor, this distinction is consequential: one generated reply praised an incorrect answer while also giving the correct next number. Repeated sampling may yield a better reply, but it does not replace checking responses or using conversation rules to catch harmful or erroneous behavior. section 12.5.2
