A chatbot that answered yesterday’s questions may struggle when users start asking about a new service. Repeated requests to rephrase, immediate opt-outs, and transfers to human agents are signs that the experience no longer meets users’ needs. Continuous improvement means measuring what happens, fixing a specific weakness, and checking whether the change helped. The cycle continues as user behavior, business rules, and technology change. section 3.1 section 1.3
When the questions change
A question-answering bot began receiving vaccine questions it did not understand, including requests about appointments and second doses. It often asked users to rephrase; frustrated users then reached the call center instead. The team’s first task was not simply to add answers, but to agree on what improvement would mean and measure whether subsequent changes reduced these failures. section 3.1
Use a short cycle for each worthwhile problem:
- Measure current performance to establish a baseline. Choose a problem tied, where possible, to a user or business outcome. section 1.3.2
- Implement a focused fix, such as updating one poorly performing intent or clarifying one dialogue step, rather than changing everything at once. section 1.3.2
- Deploy the fix, record its effect on the original problem, and repeat. Keep a successful change; undo or revise one that makes things worse. section 1.3.2
Figure 1.8 Cumulative success in a process is dependent on success in each of the individual steps. Visually it looks like a funnel that narrows after each step.
The funnel makes a weak link visible: users must remain engaged, be understood, and receive what they need. Losses at each step reduce overall success, so improving only the final answer may not help users who never reach it. section 1.3.1
Choose metrics that fit the bot’s job. A routing agent needs to send users to the right destination; a process-oriented bot needs to help them reach a goal; a question-answering bot needs prompt, accurate responses without unnecessary follow-ups. Outcomes may include containment, time to resolution, customer satisfaction, or calls routed to the wrong human. Explain a proposed intent-recognition fix in those terms, not only as a classifier score. section 3.0 section 1.3.3
Prioritize problems by frequency, expected benefit, fix complexity, and team capacity. Fixed-duration sprints can organize the next deliverable changes, but deployment is not proof of success: analyze outcomes and run regression tests. Small changes limit the scope of unexpected effects and make it easier to learn what worked. Generative AI does not remove the need for this work; it can still misunderstand users or give wrong answers. section 3.6 section 1.3.2 section 1.3.1
