A task success metric measures whether users accomplish an intended goal, rather than merely signing up or spending time in a chatbot. For an educational chatbot, the central question is whether students master the targeted skills. section 12.7.3
Useful measures include:
- Summative assessment: automatically graded quizzes or tests that measure students’ abilities. section 12.7.3
- Formative assessment: questions embedded in lessons to measure progress during learning. section 12.7.3
- Lesson outcomes: completion percentage, time per lesson, partial progress, and correct use of mathematical terms. section 12.7.3
Math tutor chatbot
Evaluate success by tracking whether students complete lessons, improve their skills, solve math riddles, or use mathematical expressions correctly—not only by counting sessions or questions. section 12.7.3
Figure 12.7 Sankey diagram of lesson funnels
Use the lesson-funnel Sankey diagram to inspect where learners complete or abandon educational chatbot flows. section 12.7.3
