A user answers “yes,” but the chatbot does not recognize it as a valid response and repeats its question. The user says “yes” again, then abandons the exchange or asks for help. Looking at either message alone misses the failure. A dialogue is the sequence of user messages and bot responses that shows what happened across the interaction. section 3.4.1 section 4.4.1
Recorded dialogues, often called conversation logs or transcripts, help teams find problems and measure whether a chatbot understands users. A practical log layout puts each user input and bot output in one row, with rows grouped by conversation ID. Pairing a transcript with direct user feedback can clarify why an exchange was frustrating rather than leaving the team to infer it from messages alone. section 4.4.1 section 3.4.1
Choose how much of each dialogue to review based on the job:
- For a manageable intent test set, start with the first user utterance from each conversation, then remove duplicates. This often captures the primary goal, but it can miss a request made later. section 4.4.1
- For accuracy measurement, review sampled user utterances against the bot’s returned intents and have a person annotate the correct intent. Compare results with actual intent volumes before setting improvement priorities. section 4.2.1 section 4.5.1
- For a handoff summary, retain a transcript and define what a useful summary must contain. An LLM can help produce that summary, but it needs those inputs and a clear standard for the output. section 12.3
Avoid treating a sampled opening message as the whole conversation. Use full exchanges when the problem depends on earlier turns, such as a repeated question or an ambiguous answer. Check findings against user feedback and measured outcomes, then revisit the logs as user behavior changes. section 3.4.1 section 4.4.1
