liveBook · Manning

Cascade of trust

As explained in AI Governance

A cascade of trust is a failure in which successive agents treat uncertain findings from earlier agents as established facts.

A cascade of trust occurs when an agent passes on a tentative finding and the next agent treats it as fact. Uncertainty disappears at each handoff: a tentative finding becomes “the policy says,” or a guess about intent becomes the basis for action. section 6.7

Errors compound across rewrites

In one study, GPT-3.5 repeatedly rewrote news articles under different stylistic prompts. After eight rewrites, correct answers about the articles fell from about 96% to under 60%, illustrating how successive processing can compound errors. section 6.7

Table 6.6 How agentic AI amplifies existing ethical and trustworthiness risks
Chapter 6 risk
Non-agentic failure
What changes in agentic systems
Accuracy and hallucinations
The model states a made-up fact, clause, case, or source. A person reads it and decides whether to trust or use it.
The model treats its own invented fact as true and builds on it, passing it forward or storing it as memory that resurfaces in official documents.
Robustness
A typo, an odd phrasing, or a dialect changes the wording of the answer, and the change is visible in the text a person reviews.
The same small change makes the model pick a different action: different documents, a different tool, or skipping the escalation step entirely.
Consistency
Two similar customers get differently worded answers to the same question.
Two similar customers are routed differently: one gets extra checks, the other a fast decision. The gap is in treatment, not wording.
Bias and fairness
The model gives weaker answers for some groups or repeats bias from its training data. Even when it pulls documents first, the bias lands in the one answer a reviewer can see.
Bias compounds across steps: which claims the agent retrieves, which it treats as the priority case, and which action it takes. No single step looks wrong, so the pattern is hard to trace.
Overreliance
A human reviewer may negligently accept a faulty AI output which causes harm.
The reviewer sees only the finished result, not the steps behind it. By the time they look, the action is already taken.[129]
Transparency and explainability
The organization cannot trace which documents shaped an answer, what is the confidence of that answer, and whether the answer was reviewed by a human.
The organization cannot trace how the agent planned the work, which tools it used, or which resources it accessed. The model chose most of it itself.
Safety and toxicity
When prompted, the model produces harmful, manipulative, or inappropriate answers, and a person sees it before it goes anywhere.
The model bypasses HITL, for example  having a subagent approve its own action and then use that approval as if it were from a human[130].
 
IP and provenance
Carelessly providing licensed data as input to a model or using the model’s outputs can damage an organization’s intellectual property.
The agent may edit resources, combine multiple resources, or share documents autonomously, threatening intellectual property and muddying ownership.

Compare how an invented fact can be treated as true and passed forward or stored as memory in an agentic system. section 6.7

How AI Governance covers it

How deep it goes
Defines how handoffs turn tentative findings into confident inputs and illustrates compounding errors, without specifying controls.
Examples
Uses policy claims, inferred intent, and repeated GPT-3.5 article rewrites to illustrate false certainty and compounding errors.
What you can do after
Evaluate how agent handoffs can convert uncertain findings into confident inputs for later decisions.
How it connects
Connects to multi-agent systems, hallucinations, and accuracy.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.