liveBook · Manning

Inconsistency

As explained in AI Governance

A failure in which a system produces different outputs under comparable conditions, such as giving different answers to the same question.

Inconsistency occurs when different users receive different answers to the same question on different days, even when the answers draw on the same policy documents. Unlike robustness failures caused by phrasing variation within one session, this cross-user variation can be harder to detect. section 6.1.2

Treat inconsistency as a risk to reduce and make traceable:

  • Use layered controls to lower how often failures occur. section 6.8
  • Log system behavior so failures can be reconstructed when they occur. section 6.8

Different answers to the same policy question

Two users ask the same question on different days and receive different answers from a system using the same policy documents. This is a consistency problem, not merely a response to different phrasing. section 6.1.2

How AI Governance covers it

How deep it goes
Defines the failure and notes its detection challenge, but does not detail a specific consistency-control procedure.
Examples
Uses the case of different users receiving different answers on different days from the same policy documents.
What you can do after
Diagnose failures through logs and reduce their frequency with layered controls.
How it connects
Connects inconsistency with robustness, hallucinations, layered controls, and logging.

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.