liveBook · Manning

Consistency

As explained in AI Governance

The degree to which a system produces the same outputs under comparable conditions.

Consistency means giving equivalent cases the same answer. Generative AI can vary its conclusions because it predicts tokens rather than following fixed rules; wording, chat history, task length, provider updates, or new training data can contribute to variation. In consequential settings, inconsistent answers create complaints, weaken trust, and complicate audits. section 6.1.3 section 6.1

ClaimAssist courtesy-car eligibility

Two customers with similar accidents ask whether their policies include a courtesy car. ClaimAssist gives one a clear yes and the other a no, despite accessing the same documents and both policies including the benefit. The team routes the decision through a deterministic rule engine; rerunning the transcripts gives both customers the same eligibility answer. section 6.1.3

Consistency controls can include:

  • Changing model configuration; setting temperature to zero is given as an example. section 6.1
  • Using structured output formats to reduce variation, or delegating decisions to deterministic rules. section 6.1.3 section 6.1
Figure 6.2 The three faces of unreliability in GenAI. The same ClaimAssist query produces three distinct failure modes: hallucination, where the model invents a policy clause that does not exist; robustness failure, where a minor typo flips the recommendation; and consistency failure, where two identical queries at different times return opposite answers.

Compare consistency failures with hallucination and robustness failures in the same ClaimAssist scenario. section 6.1

How AI Governance covers it

How deep it goes
Explains causes and operational risks, then demonstrates deterministic decision routing and names configuration controls.
Examples
Uses opposing courtesy-car eligibility answers for two customers whose policies both include the benefit.
What you can do after
Apply a deterministic rule engine to eligibility decisions instead of leaving them to variable model outputs.
How it connects
Distinguishes consistency from hallucination and robustness as separate reliability problems.

Also explained in

Compare how 28 books explain consistency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.