LLM response quality can decline gradually after model updates, prompt changes, or shifts in user behavior. Because responses may still sound reasonable, manual checks can miss systematic problems until they affect user satisfaction or trust. section 10.5.3 section 10.5
Monitor for quality changes with complementary checks:
- Run a curated set of known-good questions regularly, checking for required information and response constraints. section 10.5.3
- Score response characteristics automatically and alert when quality patterns change. section 10.5.3
- Compare candidate models or prompts with production responses through asynchronous shadow tests before deployment. section 10.5.3
Support responses
A password-reset check can flag answers that omit the email link or exceed a length limit. A model update that causes these failures to appear more often signals a quality change worth investigating. section 10.5.3
GOLDEN_TESTS = [ #1
{
"question": "How do I reset my password?",
"expected_elements": ["email link", "account settings", "24 hours"],
"quality_criteria": {
"max_length": 200,
"must_include": ["reset", "password"],
"tone": "helpful"
}
},
{
"question": "What's your refund policy?",
"expected_elements": ["30 days", "original payment method"],
"quality_criteria": {
"max_length": 150,
"must_include": ["refund", "policy"],
"tone": "professional"
}
}
]
def run_quality_check(llm): #2
results = []
for test in GOLDEN_TESTS:
response = llm.generate(test["question"])
score = evaluate_response(response, test["quality_criteria"])
results.append({"test": test["question"], "score": score})
return results
Example golden tests specify expected answer elements and criteria such as maximum length, required terms, and tone. section 10.5.3
