A failure mode is a recurring way a model or system breaks under particular conditions. Overall quality can hide failures that cluster by task, user segment, or operating condition. section 3.4.5 section 7.6.1
Look for patterns such as:
- A support assistant that hallucinates more on policy exceptions or gives weaker answers in some languages. section 3.4.5
- An agent that struggles when a tool times out, returns partial data, or a task takes many steps. section 3.4.5
- An LLM judge affected by position, verbosity, style, insufficient context, prompt injection, or overconfidence. section 9.7
From offline signal to live watchpoint
A recommendation model repeatedly puts Batman first for users without relevant watch history. Track repeat-title frequency in top recommendations by user segment; alert when it exceeds the control by more than 15% for two consecutive days. The signal can support a decision to pause, roll back, or test a patched model. section 7.6.1
Identify failure patterns in offline evaluation, translate each into an equivalent live metric, and set thresholds for alerts. Keep those watchpoints alongside experiment success metrics so known risks can be caught before an A/B test ends. section 7.6 section 7.6.1





