AI system development can run as a repeating loop: define what good behavior means, measure the system against those criteria, analyze failures, and apply changes to close the gap. Evaluation and alignment are interdependent: measurement reveals what needs improvement, and alignment acts on that signal. section 1.3.1 section 1.3
Use the cycle to turn observed behavior into targeted changes:
- Define measurable criteria for the desired behavior. section 1.3.1
- Test the system and measure each criterion. section 1.3.1
- Analyze errors to identify patterns and failure modes. section 1.3.1
- Apply an alignment intervention, then evaluate again. section 1.3.1
Customer-service chatbot
For an e-commerce chatbot, criteria include order-status accuracy, politeness, response time, and issue resolution. Testing 500 conversations reveals 87% accuracy and hallucinated shipping dates when database values are null. A prompt change and retrieval of shipping-policy documents address the failure; re-evaluation raises accuracy to 94% and reveals a new issue to investigate. section 1.3.1
Figure 1.10 The iterative cycle of AI engineering. Define establishes what success looks like, Evaluate measures the current system against the designed evaluation metrics, Error analysis identifies failure modes, and then Align optimizes the system using different methods. The cycle then repeats.
The cycle diagram traces the progression from defining success through evaluation and error analysis to alignment, then repeats. section 1.3.1










































