Semantic evaluation tests whether an LLM response preserves the intended meaning instead of requiring an exact string match. It is necessary because identical prompts can produce different but equally valid outputs. section 13.2 section 12.1.1
Evaluate responses against qualities that remain meaningful when wording changes:
- Semantic correctness: determine whether the response conveys the intended meaning. section 13.2
- Factual accuracy: check whether claims are correct rather than merely well-formed. section 13.0
- Guideline adherence: verify that responses stay appropriately scoped and aligned with business policies. section 13.0
RAG documentation assistant
A documentation assistant may answer the same engineering question with different wording across runs. Semantic evaluation accepts variation when the answer remains accurate, appropriately scoped, and aligned with the intended guidelines. DeepEval provides a named implementation path for semantic evaluation in production LLM applications. section 1.4.2 section 13.0
Listing 12.1 Parameters that control output randomness
temperature=0.0 #1 temperature=0.7 #2 temperature=1.5 #3 top_k=1 #4 top_k=40 #5 top_k=100 #6 top_p=0.1 #7 top_p=0.9 #8 top_p=1.0 #9
Use sampling parameters to understand why repeated calls vary and why evaluation must judge meaning rather than exact output text. section 12.1.1
