Error analysis systematically examines model failures to find recurring causes and patterns, then uses those findings to reduce the gap between current and desired behavior. section 8.5.1 section 1.3.1
Use error analysis as a practical diagnostic loop:
- Measure the system against explicit criteria such as accuracy, politeness, latency, or resolution. section 1.3.1
- Inspect incorrect outputs and group them into recurring failure modes. section 1.3.1
- Apply a targeted alignment change, such as revising a prompt, adding retrieval, or extending a specification. section 8.5.1 section 1.3.1
- Re-evaluate the modified system and investigate newly exposed errors. section 1.3.1
Null shipping information
A customer-service chatbot achieves 87% factual accuracy, but analysis of its inaccurate responses reveals hallucinated shipping dates when the database returns null values. The system is changed to state that information is unavailable and to retrieve shipping-policy documents; accuracy then rises to 94%, exposing a different issue for the next cycle. section 1.3.1
Figure 1.10 The iterative cycle of AI engineering. Define establishes what success looks like, Evaluate measures the current system against the designed evaluation metrics, Error analysis identifies failure modes, and then Align optimizes the system using different methods. The cycle then repeats.
The iterative-cycle figure situates error analysis between evaluation and alignment, showing why diagnosis and re-evaluation repeat throughout AI system development. section 1.3.1



