A chatbot may answer confidently but send a user down the wrong path. If it labels an accident report as a request to speak with an agent, the user does not get the intended help. Precision measures how often a selected intent is actually correct: correct selections divided by all selections of that intent. Low precision means the classifier is over-selecting it. section 5.1.1 section 4.2.1
Six requests for an agent?
In a blind test, the classifier selected the request-for-an-agent intent six times. Four utterances really did request an agent; two concerned other topics. Its precision was therefore 4 ÷ 6, or about 0.66. A separate agent request was missed: that miss affects recall, not this precision calculation. section 5.1.1
Table 5.1 Test results show seven utterances, five of which are labeled with the correct #Request_Agent intent. The first four predictions were true positives. The last two rows show where #Request_Agent was predicted twice for utterances where it shouldn’t have been (“What can I ask you” and “Somebody hit my car”). These false positives contribute to our precision calculation: 4 / (4 + 2) = 0.66.
|
Utterance
|
Correct intent
|
Predicted intent
|
Correct
|
|---|---|---|---|
|
Customer service
|
Request_Agent
|
Request_Agent
|
1
|
|
Speak with an agent
|
Request_Agent
|
Request_Agent
|
1
|
|
Can I please speak with somebody?
|
Request_Agent
|
Request_Agent
|
1
|
|
Talk with a human
|
Request_Agent
|
Request_Agent
|
1
|
|
When will I get a live person?
|
Request_Agent
|
Office_Hours
|
0
|
|
What can I ask you?
|
VA_Capabilities
|
Request_Agent
|
0
|
|
Somebody hit my car
|
Report_Accident
|
Request_Agent
|
0
|
Compare each utterance’s correct intent with its predicted intent to see the four correct selections and two false positives behind the score. section 5.1.1
Precision answers a different question from recall. Precision asks, “When this intent was predicted, how often was it right?” Recall asks, “Of the utterances that belonged to this intent, how many were found?” Overall accuracy gives a quick view of correctness but can hide which intents are being over- or under-selected. Review precision alongside recall and F1 score rather than relying on one number. section 5.1.1
To investigate low precision, inspect the wrong selections in blind-test results and compare them with the training examples. One intent about the virtual assistant had perfect recall but only 0.50 precision: the classifier selected it for a question about missing tags. Removing redundant or unsupported training examples raised its precision in the next test. The test contained only one true example of that intent, however, and another intent’s recall fell, so the new score was not a reason to stop checking. section 5.2.2
The same idea applies to retrieval: precision is the proportion of retrieved documents relevant to the user’s query. Balance it with recall so a chatbot does not trade away useful information merely to avoid irrelevant results. For intent classifiers, keep testing with representative user utterances when available, then monitor and retrain as usage changes. section 6.5.2 section 5.5
