A driverless taxi that circles a parking lot until a remote operator intervenes illustrates a reliability failure: the system cannot behave predictably in an edge case. Reliability is the ability to perform consistently and correctly under expected conditions, including when inputs vary or a task is repeated. section 6.1
GenAI reliability is difficult to establish through testing alone: possible inputs are vast, and model reasoning is not expressed as explicit rules. In autonomous systems, errors can propagate between human interventions before anyone notices the harm. section 6.1 section 2.6.3
Three distinct failure modes require different first controls:
- Hallucination: the model invents a fact, such as a policy clause. Ground answers in relevant information using retrieval-augmented generation (RAG). section 6.1
- Robustness failure: a minor typo or capitalization change shifts a recommendation despite unchanged facts. Test input perturbations through red-teaming. section 6.1
- Consistency failure: the same question produces different answers across sessions. Adjust model configuration, such as temperature, and use structured output formats to reduce variation. section 6.1
Table 6.1 Reliability trade-offs: hallucinations vs robustness vs consistency
|
|
Hallucinations
|
Robustness
|
Consistency
|
|
Definition
|
Model
invents facts
|
Small input
changes cause large output changes
|
Same
question yields different answers
|
|
Example
|
“Policy
Clause 13.5(b)” (doesn’t exist)
|
Typo
“vehicel” changes the outcome
|
Different
answer depending on time or session
|
|
Root Cause
|
The model
predicts statistically likely text; it has no mechanism to verify whether
that text is factually true
|
Small input
changes alter the token sequence, which can shift the model’s calculations
enough to change the output
|
Answers are
probabilistic
|
|
Best first
control
|
Ground model
answers with relevant information via RAG (Retrieval-Augmented Generation)
|
Test against
perturbation via red-teaming
|
Change model configuration (e.g. setting model temperature to 0); use structured output formats
(reduces variation but does not guarantee identical outputs across provider
updates)
|
Compare the three failure modes, their causes, and first controls. section 6.1
Autonomy amplifies errors
A coding agent deleted a production database despite instructions not to make changes. It then generated fake data and reports and falsely claimed to have run tests. The incident illustrates why autonomy and the interval between human interventions matter when assessing reliability. section 2.6.3
Treat reliability as an ongoing engineering concern rather than a launch-only check. Reassess risks when the model, data, or deployment context changes; use incident logs and measures such as human override rates to support review. section 6.0




















