liveBook · Manning

Postmortem

As explained in AI Model Evaluation

A postmortem is a blameless retrospective analysis of a production or evaluation incident that records what happened, why detection or validation failed, and what changes will follow.

A postmortem is a blameless retrospective analysis of a production or evaluation incident. It records what happened, what actually broke, why detection or validation failed, and what will change. The goal is not merely to fix the immediate problem, but to shorten detection time and improve confidence in future decisions. section 10.6 section 10.6.3

Run an evaluation postmortem shortly after triage, while the incident context remains fresh:

  • Describe what happened and identify what actually broke, such as a stale rubric, drifted judge model, data-pipeline issue, or genuine model regression. section 10.6.3
  • Explain why validation did not catch the problem sooner instead of treating the first visible symptom as the whole failure. section 10.6.3 section 10.6.4
  • Record what changes as a result, keeping the discussion blameless and focused on system improvement. section 10.6.3
  • Triangulate judge scores with deterministic checks, standing human review, online product metrics, cross-model panels, canary sets, and stability tests so signal divergence exposes quiet failures. section 10.6

Restoring trust after an evaluation failure

After correcting a drifted judge or stale rubric, re-run the full validation loop against the gold-standard dataset rather than checking only the triggering case. Then communicate what was wrong, how long it affected decisions, what changed, and how validation will become more frequent. A concise, specific one-pager gives product managers, engineers, and leadership evidence for renewed launch or go/no-go decisions. section 10.6.4

How AI Model Evaluation covers it

How deep it goes
Defines four review questions, adds signal triangulation, and extends recovery through full re-validation and clearer communication.
Examples
Uses stale rubrics, drifted judge models, data-pipeline issues, model regressions, gold-standard datasets, and corrective one-pagers.
What it assumes
Assumes an evaluation system with validation data or signals, while treating failure response as a general engineering practice.
What you can do after
Diagnose evaluation incidents, document causes, re-run validation, communicate changes, and strengthen monitoring after recovery.
How it connects
Connects evaluation failure with validation, deterministic checks, human review, online metrics, canary sets, and stability testing.

Also explained in

Compare how 5 books explain postmortem →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.