A user asks a chatbot for help, gets an unrelated answer, rephrases the question, and eventually asks for a human agent. Accuracy gives a way to measure this failure: it is the proportion of evaluated predictions or answers judged correct. It describes observed performance, not a guarantee that the next answer will be right. section 4.1.1 section 4.5
Reading an accuracy score
If 79 of 100 tested questions receive the correct intent prediction, overall classifier accuracy is 79%. That quick summary does not reveal which intents are being missed or incorrectly selected. Examine recall, precision, and F1 score to find those patterns. section 5.1.1
What counts as correct depends on the job. A classifier must predict the right intent so the chatbot can deliver the appropriate response or process flow. A generative solution must produce a correct, useful answer. For a process-oriented bot, judging whether it takes the correct action also matters. section 4.2 section 4.6
Choose a test that fits the bot’s stage and available data:
- Before deployment, use k-fold cross validation when only training examples are available. Its score mainly tests whether those examples were labeled consistently. section 4.2.1
- Once real interactions are available, judge predictions against a representative blind test set drawn from user logs. Divide correct judgments by all candidates judged to establish a baseline. section 4.2.1 section 4.5
- Track accuracy after changes, then inspect high-volume, poorly performing intents rather than treating the overall percentage as the whole story. section 4.5.1 section 5.5
Figure 4.5 K-fold cross validation and generated test data are suitable for situations where representative data is not available. Once a solution is deployed to production, representative blind test data will produce the most reliable measurement of your classifier’s ability to understand.
This figure helps distinguish tests made from existing or generated data from representative blind tests using real user utterances. section 4.2.1
Accuracy can improve without becoming a promise. Adding real examples of password resets, lockouts, and security codes raised one classifier’s overall score from 76% to 79%. Later, adding an intent for newly appearing cancellation questions produced 92% on an updated blind test. Keep monitoring: new user needs and topics can change what the bot must recognize. section 5.2.1 section 5.3.1
