A sampling strategy selects generated tokens from an LLM’s probability distribution. It introduces controlled randomness, making responses more natural and varied but also nondeterministic. section 12.1.1
Configure sampling with parameters that trade deterministic behavior against variety:
- Temperature controls randomness: 0 selects the most probable token, while higher values such as 0.7–1.0 increase variety and may reduce coherence. section 12.1.1
- Top-K restricts each step to the K most probable tokens; top_k=1 is deterministic, while top_k=40 permits more variation without considering very unlikely tokens. section 12.1.1
- Top-P, or nucleus sampling, dynamically adjusts the candidate vocabulary according to the probability distribution rather than using a fixed K. section 12.1.1
Listing 12.1 Parameters that control output randomness
temperature=0.0 #1 temperature=0.7 #2 temperature=1.5 #3 top_k=1 #4 top_k=40 #5 top_k=100 #6 top_p=0.1 #7 top_p=0.9 #8 top_p=1.0 #9
Compare representative temperature, top-K, and top-P settings, from deterministic choices to highly variable generation. section 12.1.1
Because sampling can produce multiple valid responses for one prompt, exact-string assertions are insufficient. Evaluate semantic correctness, factual accuracy, and adherence to guidelines instead. section 12.1.1
