A postmortem is a blameless retrospective analysis of a production or evaluation incident. It records what happened, what actually broke, why detection or validation failed, and what will change. The goal is not merely to fix the immediate problem, but to shorten detection time and improve confidence in future decisions. section 10.6 section 10.6.3
Run an evaluation postmortem shortly after triage, while the incident context remains fresh:
- Describe what happened and identify what actually broke, such as a stale rubric, drifted judge model, data-pipeline issue, or genuine model regression. section 10.6.3
- Explain why validation did not catch the problem sooner instead of treating the first visible symptom as the whole failure. section 10.6.3 section 10.6.4
- Record what changes as a result, keeping the discussion blameless and focused on system improvement. section 10.6.3
- Triangulate judge scores with deterministic checks, standing human review, online product metrics, cross-model panels, canary sets, and stability tests so signal divergence exposes quiet failures. section 10.6
Restoring trust after an evaluation failure
After correcting a drifted judge or stale rubric, re-run the full validation loop against the gold-standard dataset rather than checking only the triggering case. Then communicate what was wrong, how long it affected decisions, what changed, and how validation will become more frequent. A concise, specific one-pager gives product managers, engineers, and leadership evidence for renewed launch or go/no-go decisions. section 10.6.4




