liveBook · Manning

Continuous improvement

As explained in Designing AI Systems

An asynchronous improvement loop evaluates completed interactions after serving them and feeds the resulting scores into systematic system improvement.

Continuous improvement turns production interactions into evidence for system changes. The serving path returns a response first; afterward, completed traces can be scored and their results used to guide systematic improvement. section 7.7 section 8.5

A repeatable improvement cycle connects production behavior to controlled changes:

  • Collect production traces and attach quality signals from automated, LLM-based, or human scoring. section 8.0
  • Move low-scoring or edge-case traces into evaluation datasets, then compare candidate changes offline. section 8.0
  • Test promising variants in production with A/B testing, then use the results to inform the next cycle. section 8.0
Figure 8.5 Asynchronous scoring and experiment feedback loop for a patient intake request. After the response is returned, scoring rules evaluate the completed trace and record results. If an A/B test is active, the experimentation service associates those outcomes with the assigned variant to support comparison and improvement.

The asynchronous path scores a completed patient-intake trace after the response is returned, records the result, and associates it with an experiment variant when applicable. section 8.5

How Designing AI Systems covers it

How deep it goes
Details a cycle from trace scoring through dataset curation and offline evaluation to A/B testing, without specifying scoring implementations.
Examples
Uses completed patient-intake traces to illustrate asynchronous scoring and experiment-variant comparison.
What it assumes
Builds on production traces and quality scores attached to interactions.
What you can do after
Operate a feedback cycle that scores live traffic, compares alternatives, and feeds results into production decisions.
How it connects
Links observability, quality scoring, evaluation datasets, offline evaluation, A/B testing, and human annotation.

Also explained in

Compare how 43 books explain continuous improvement →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.