A chatbot can appear inaccurate when its test examples have not been judged consistently. Data annotation assigns each clear user utterance the correct intent, creating a dependable basis for training or evaluation. The difficult cases are not unintelligible inputs but valid utterances whose goals overlap, such as requests for an agent and questions about office hours. section 4.4.4
A practical annotation workflow is:
- Narrow the data to utterances expressing a clear goal in the system’s domain, and scrub them where appropriate. section 4.4.4
- Apply the current intent criteria rather than dividing examples mechanically among annotators. section 4.4.4
- Have annotators discuss judgments together or independently label overlapping samples, then resolve disagreements. section 4.4.4
- When runtime predictions are logged, verify correct predictions first and label the remaining incorrect utterances. section 4.4.4
Building a golden test set
Label “I want to speak with a real person,” “Can I talk to a manager,” and “Get me customer service” as Request_Agent. Label “Are you open on Sundays,” “What time do you open,” and “When does your office close” as Office_Hours. Each utterance-intent pair becomes a row in a test set; response evaluation can instead pair an utterance with its expected answer, such as baggage rules or refund timing.
Figure 4.5 K-fold cross validation and generated test data are suitable for situations where representative data is not available. Once a solution is deployed to production, representative blind test data will produce the most reliable measurement of your classifier’s ability to understand.
Use the evaluation-method comparison to select annotation-heavy representative blind testing after deployment, while using other methods when representative data is unavailable. section 4.2.1
