A chatbot can return a polished answer that misses the user’s need. Understanding is its ability to identify what the user wants and respond appropriately; weak understanding appears through wrong answers, complaints, declining engagement, rising abandonment, and avoidable escalations. section 4.0
Measure understanding according to the system’s approach. For a classifier, measure how accurately it predicts the user’s intent. For a generative model, assess whether its output is correct and useful. The method also depends on whether the system is traditional, generative, or hybrid, and whether it is predeployment or postdeployment. section 4.2
Improve understanding through an iterative, evidence-based process:
- Assess the current state with prepared log data, a baseline, and a test set that assigns each utterance a golden intent. section 4.0 section 5.0
- Identify classifier error types, prioritize problems with data-science methods, and repeat improvement cycles rather than treating performance as permanent. section 5.0 section 4.0
- Use generative AI to suggest vocabulary and test data while keeping the human builder responsible for decisions and truthfulness. section 7.0
Password-reset intent
An IT help-desk assistant needs strong understanding of password resets. Users may say “forgot password” or “cannot login,” but their real vocabulary is broader. Generative AI can help imagine additional utterances when production examples do not yet exist; the human builder reviews and approves them. section 7.0
