A chatbot that treats “I can’t get on my profile” as an unrelated request leaves the user stuck, even if it recognizes “password reset.” A training dataset is the collection of examples used to fit its intent classifier. Each example helps the classifier connect a user utterance—what someone says—to an intent—what they want. If the examples miss the language users actually use, the bot may choose the wrong intent or fall back to a generic response. section 1.1.3 section 5.2.1
Build and review the dataset around user needs, not just example counts:
- Start with historical chats, call-center transcripts, or emails when available. Check that examples represent the intended topics and use varied, meaningful phrasing. section 1.1.3
- Inspect example counts and overlap across intents. A few examples for one intent and hundreds for another can contribute to wrong selections; count alone does not establish the cause of an accuracy problem. section 4.3.1 section 4.5.1
- Compare the training mix with annotated user logs and accuracy for each intent. Prioritize important, frequent requests the bot misses, and look for in-domain requests outside its current scope. section 4.5.1
Make a login intent less brittle
Four login examples focused on phrases such as “online account problem.” They missed requests about profiles, lockouts, password resets, and security codes. Adding representative log utterances—including “Help signing in to online portal” and “I need a security code to log on”—raised login-intent recall on a blind test from about 44% to 89%; overall accuracy rose from 76% to 79%. The test utterances stayed separate from training data, so improvement was checked on requests the classifier had not learned from. section 5.2.1 section 7.1.3
When logs reveal an entirely new topic, create an intent rather than forcing those requests into an old one. Cancellation requests, for example, were divided into nine training utterances for a new intent and thirteen blind-test utterances. A later data-breach intent used varied terms such as “hack,” “breach,” and “stolen.” Recheck other intents after each change: an overall score can hide a newly weakened topic. section 5.3.1
When real examples are scarce, an LLM can suggest initial utterances, missing vocabulary, synonyms, and grammatical variations. Use domain knowledge and available real-world utterances to guide it, then select or refine useful output and test the classifier. Prefer a small set covering distinct ways to ask over many near-duplicates of one phrase. Keep testing data separate, and iterate rather than assuming the first generated set is good enough. section 7.1.1 section 7.1.3 section 7.2
Table 7.2 Comparing low-variation to high-variation utterances. High-variation utterances increase the robustness of classifiers.
|
Low-variation utterance set
|
High-variation utterance set
|
|---|---|
|
• I forgot my password
• Forgot my password
• Forgot password
• Help I forgot my password
|
• I can’t log in
• Account locked out
• Forgot password
|
Compare repeated versions of “forgot password” with the shorter, more varied set that also includes “I can’t log in” and “Account locked out.” section 7.1.3
A training dataset also matters when choosing an existing LLM. Review its model card, when available, for training-data and methodology information relevant to your use; dataset selection is one guardrail, not a substitute for other safeguards. For a chatbot’s own classifier, keep auditing the training set as user requests and the topics they expect it to handle change. section 1.2.2 section 4.2.3 section 5.3.1
