A chatbot may answer familiar training examples correctly yet misunderstand how real users ask for help. An evaluation dataset—often called a blind test set—checks its behavior on inputs kept separate from training. When those inputs have reviewed, expected intents or answers, they form a golden test set that can be reused to compare changes against a baseline. section 7.1.3 section 4.6
Build a useful set from real conversations when logs are available:
- Sample across weeks or months so seasonal topics and events are not missed. A first user turn is often a useful shortcut, but important requests can occur later. section 4.4.1
- Sort candidate utterances before labeling. Set aside unusable inputs, check whether a request is in scope, and scrub personal information where needed. section 4.4.2 section 4.4
- Annotate each retained input for the task: assign the correct intent for a classifier or an ideal response for generative AI. Resolve disagreements about overlapping intents rather than silently dividing the labeling work. section 4.4.4
- Keep test utterances out of training. If logs reveal a new intent, put some examples into training and others into the blind test set. section 7.1.3 section 5.3.1
Figure 4.7 Once you obtain some data, each utterance should first be sorted into buckets to identify potential candidates; this will separate the good, usable test data from the bad or irrelevant user inputs. The data may also need to be scrubbed to fix problems like personal identifiable information (PII). After that, the data will need to be annotated (for classifiers, it will need to be labeled with the correct intent; for generative AI, it will need to be associated with an ideal output response). Finally, the data will need to be converted into one or more sets that can be consumed by your testing tool.
The preparation path runs from candidate selection and privacy scrubbing through annotation to a format a testing tool can consume. section 4.4
Check a change without guessing
Suppose users say “I can’t get on my profile” or ask for a security code, while a classifier has mostly learned examples phrased as login problems. A blind test exposed missed login requests. After representative training examples were added, rerunning the set showed login-issue recall rising from about 44% to 89%, while precision remained steady. The held-out requests made the improvement measurable rather than anecdotal. section 5.2.1
For generated answers, an expected response is a reference, not the only acceptable wording. Review whether an answer is correct, sufficient, appropriate, and safe—including whether it avoids sensitive disclosures and prompt-injection failures. The annotated set can then help track changes to prompts or model settings. section 4.2.2
Refresh the dataset as user needs and the chatbot change. New intents, merged intents, expanded knowledge-base coverage, and new prompt scenarios call for corresponding tests. Review logs regularly: a set that once represented users may miss questions prompted by a new product, policy, or data breach. section 4.5.3
