Monitoring is the ongoing check that a deployed AI system works for users, not merely that its server stays up. A support chatbot can have 99.9% uptime, fast responses, and no HTTP errors while confidently giving customers an incorrect refund policy for weeks. Track technical health alongside answer quality, user experience, and cost. section 10.3 section 10.3.1
Organize the signals around four operational questions:
- Can users get help quickly? Measure availability, response time, and infrastructure performance. section 10.3.1
- Are answers useful? Check for incorrect, harmful, or unsupported content; track refusal rates, repetition, response length, and unexpected language changes. section 10.3.6
- Do users leave satisfied? Combine voluntary ratings and comments with follow-up questions and handoffs to human support. section 10.4 section 10.4.1
- Is the system affordable? Compare hourly spend with a historical baseline and watch for rising trends after prompt changes, traffic shifts, or model switches. section 10.3.4
Use layers rather than one quality score. Simple response checks can flag empty answers, unnecessary refusals, and leaked error messages. Statistical measures can reveal drift. An LLM judge can assess an answer against a question or reference text, while user feedback tests whether a technically plausible answer actually helped. Quality is context-dependent, so automated signals warrant investigation rather than automatic claims of correctness. section 10.3.6 section 10.4
Recurring hallucination check
A response says a product has a three-year warranty, but its reference says two years. Run a judge over a sample of real responses on a recurring schedule, retain its structured label and explanation, track the hallucination rate, and alert when it exceeds a chosen threshold. Sampling makes this workable when manually reviewing thousands of interactions is impractical. section 9.1.6
from phoenix.evals import (
HALLUCINATION_PROMPT_TEMPLATE,
HALLUCINATION_PROMPT_RAILS_MAP,
OpenAIModel, llm_classify
)
import pandas as pd
df = pd.DataFrame({
'reference': ['Product has 2-year warranty...'],
'response': ['The product comes with a 3-year warranty...']
})
model = OpenAIModel(model_name='gpt-5.4')
rails = list(HALLUCINATION_PROMPT_RAILS_MAP.values())
results = llm_classify(
dataframe=df,
template=HALLUCINATION_PROMPT_TEMPLATE,
model=model, rails=rails,
provide_explanation=True
)
This Phoenix evaluation sketch passes reference-and-response rows through a hallucination prompt, constrains the judge to category labels, and requests explanations. Its DataFrame output can feed a recurring rate calculation; the listing itself does not implement scheduling or alerts. section 9.1.6
Set up monitoring before scaling, then use trends and alerts to investigate regressions and guide updates. A drop in feedback participation can itself be an early warning; forcing ratings can worsen the experience and bias the data. Monitoring complements predeployment evaluation rather than replacing it. section 11.6.6 section 10.4.1 section 1.4.3
