Factuality measures whether generated content agrees with verifiable facts. In LLM applications, it is evaluated instead of exact string equality because identical prompts can produce different valid responses. section 13.2 section 12.1.1
Use factuality as part of an evaluation approach that:
- Assesses semantic correctness and factual accuracy rather than relying on traditional assertions such as expected-string matching. section 13.2 section 12.1.1
- Accounts for variation caused by sampling strategies, temperature, top-K, and top-P settings when interpreting results. section 12.1.1
- Combines factuality with adherence to guidelines and safety checks for production validation. section 13.2 section 13.5
Listing 12.1 Parameters that control output randomness
temperature=0.0 #1 temperature=0.7 #2 temperature=1.5 #3 top_k=1 #4 top_k=40 #5 top_k=100 #6 top_p=0.1 #7 top_p=0.9 #8 top_p=1.0 #9
Compare temperature, top-K, and top-P settings to understand how output randomness complicates factuality evaluation. section 12.1.1
