A chatbot can answer familiar questions well yet invent a plan detail or quietly degrade after a model update. An evaluation dataset makes those failures measurable: it pairs representative questions with vetted reference information or known-good response characteristics, then tests the system against them repeatedly. section 3.6.2 section 10.5.3
Build coverage around what users actually ask:
- Include factual questions, comparisons, troubleshooting, billing and account requests, and hypotheticals. A telecom test might ask for a plan price, compare allowances, or ask what happens after a data limit. section 3.6.2
- Attach current, authoritative answer sources to each query: product specifications, policies, troubleshooting guides, or FAQs. These establish the factual reference for judging a response. section 3.6.2
- Include plausible-sounding hallucinations to challenge factual assessment. For ongoing quality checks, also record essential response elements and criteria such as length, required terms, and tone. section 3.6.2 section 10.5.3
GOLDEN_TESTS = [ #1
{
"question": "How do I reset my password?",
"expected_elements": ["email link", "account settings", "24 hours"],
"quality_criteria": {
"max_length": 200,
"must_include": ["reset", "password"],
"tone": "helpful"
}
},
{
"question": "What's your refund policy?",
"expected_elements": ["30 days", "original payment method"],
"quality_criteria": {
"max_length": 150,
"must_include": ["refund", "policy"],
"tone": "professional"
}
}
]
def run_quality_check(llm): #2
results = []
for test in GOLDEN_TESTS:
response = llm.generate(test["question"])
score = evaluate_response(response, test["quality_criteria"])
results.append({"test": test["question"], "score": score})
return results
A Python golden-test listing stores questions, expected elements, and quality criteria together. Its test runner generates responses and passes them to an evaluator; the listing does not define that evaluator or require identical wording. section 10.5.3
Run the dataset regularly and before changing models or prompts. Track whether essential elements disappear or quality patterns shift, rather than expecting identical LLM outputs. For RAG changes, use robust datasets to compare configurations such as chunk sizes and embedding models. Include common and edge cases, compare results across demographic groups, and retain human review: aggregate or automated scores can miss important failures. section 10.5.3 section 3.6.7 section 11.6.6
