Accuracy can show that a classifier is usually right while hiding whether it misses important intents or predicts the wrong ones. The F1 score helps expose that balance by combining precision and recall into one classification measure. section 5.1.1
Think of recall as finding the correct intent and precision as avoiding incorrect intent predictions. The F1 score summarizes both, so improving one measure alone may not be enough. section 5.1.1
Improving an intent
For #Login_Issue, the baseline classifier predicted 5 of 9 examples correctly, producing recall of 0.4444, precision of 1, and an F1 score of 0.6153. After representative examples were added, recall rose to 0.8889, precision stayed at 1, and F1 rose to 0.9412. For #Item_Not_Received, adding 10 logged examples increased F1 from 0.56 to 0.8, although precision fell slightly. section 5.2.1 section 5.2.3
Use F1 results to guide classifier improvement:
- Prioritize high-volume intents with weak performance. section 5.5
- Add representative examples from real interaction logs, then rerun blind tests. section 5.2.1
- Inspect recall, precision, and error types rather than relying on overall accuracy alone. section 5.1.1
Table 5.2 Recall, precision, and F1 score for #Request_Agent
|
Intent
|
Recall
|
Precision
|
F1 score
|
|---|---|---|---|
Request_Agent
|
0.80
|
0.66
|
0.72
|
Compare recall, precision, and F1 for #Request_Agent, where recall is 0.80, precision is 0.66, and F1 is 0.72. section 5.1.1
