Nondeterminism means that identical inputs can produce different valid outputs across executions. In LLM systems, sampling strategies, temperature settings, and model architecture introduce this variation. section 12.1.1
Nondeterminism changes how production systems are designed and operated:
- Replace exact-output assertions with evaluations of semantic correctness, factual accuracy, guideline adherence, and safety. section 12.1.1 section 13.2
- Tune temperature, top-k, and top-p to trade response variety against coherence and predictability. section 12.1.1
- Treat variable outputs as an operational concern requiring systematic testing, monitoring, and safety controls. section 13.0 section 13.5
Controlling output randomness
A temperature of 0.0 and top_k=1 favor deterministic output. Increasing temperature to 0.7 or 1.5, or using top_k=40 or 100, allows more varied token choices. Values such as top_p=0.1, 0.9, and 1.0 adjust the probability-based candidate set. section 12.1.1
Listing 12.1 Parameters that control output randomness
temperature=0.0 #1 temperature=0.7 #2 temperature=1.5 #3 top_k=1 #4 top_k=40 #5 top_k=100 #6 top_p=0.1 #7 top_p=0.9 #8 top_p=1.0 #9
Browse the parameter listing for a compact comparison of settings that control output randomness. section 12.1.1
