liveBook · Manning

Continuous improvement

As explained in LLM Evaluation and Alignment, The Foundational Ideas

An approach to system development that repeatedly cycles through designing, measuring, and improving a system.

AI system development can run as a repeating loop: define what good behavior means, measure the system against those criteria, analyze failures, and apply changes to close the gap. Evaluation and alignment are interdependent: measurement reveals what needs improvement, and alignment acts on that signal. section 1.3.1 section 1.3

Use the cycle to turn observed behavior into targeted changes:

Customer-service chatbot

For an e-commerce chatbot, criteria include order-status accuracy, politeness, response time, and issue resolution. Testing 500 conversations reveals 87% accuracy and hallucinated shipping dates when database values are null. A prompt change and retrieval of shipping-policy documents address the failure; re-evaluation raises accuracy to 94% and reveals a new issue to investigate. section 1.3.1

Figure 1.10 The iterative cycle of AI engineering. Define establishes what success looks like, Evaluate measures the current system against the designed evaluation metrics, Error analysis identifies failure modes, and then Align optimizes the system using different methods. The cycle then repeats.

The cycle diagram traces the progression from defining success through evaluation and error analysis to alignment, then repeats. section 1.3.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Breaks the cycle into criteria, measurement, error analysis, and alignment, then demonstrates each step with a chatbot.
Examples
Uses an e-commerce chatbot with null shipping data, prompt changes, retrieval, and measured accuracy across 500 conversations.
What you can do after
Apply the cycle by defining criteria, measuring behavior, diagnosing error patterns, and choosing targeted alignment interventions.
How it connects
Connects evaluation, alignment, error analysis, prompt engineering, retrieval-augmented generation, and reinforcement learning from human feedback.

Also explained in

Compare how 43 books explain continuous improvement →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.