Variance describes how much an AI model’s responses can differ when given the same or similar input. Regular code tends to repeat the same result; a large language model produces probabilistic outputs instead. That variability can help or hurt, depending on the task. section 9.6.1
Choose the desired level of variance according to the application:
- Favor higher variance for creative tasks such as generating poetry. section 9.6.1
- Favor lower variance when evaluating a mathematical equation, where predictable responses matter more. section 9.6.1
- For generated questions, increase variety while preventing repeated questions within and across games. section 9.6.1
Figure 9.10 How temperature and top_p affect the creativity and predictability of an LLM’s output.
Compare how temperature and top_p affect the creativity and predictability of a model’s output. section 9.6.1
Model vendors expose controls for managing variance, but prompt engineering is also needed to encourage the desired behavior. Greater variance does not guarantee better answers: probabilistic generation can produce plausible but incorrect or fabricated information. section 9.6.1
