liveBook · Manning

Reliability

As explained in Vibe Engineering

Test reliability is the consistency with which tests produce correct and repeatable results for the same software and conditions.

Test reliability is the prerequisite for trustworthy CI and AI-assisted development. When tests lie, agents chase ghosts and teams lose confidence in automated feedback. section 10.0 section 10.9

Protect reliability before optimizing execution speed:

  • Remove flaky tests that produce inconsistent results. section 10.9
  • Separate circular tests that create false confidence. section 10.9
  • Prevent agents from shortcutting the oracle or gaming validation. section 10.9
  • Fix misleading tests before pursuing faster CI feedback. section 10.9

A fast liar

Sub-minute CI is not an improvement if the tests produce false confidence. A slower test suite that reports honestly is safer than a fast suite that sends agents and developers after ghosts. section 10.9

How Vibe Engineering covers it

How deep it goes
Identifies flakiness, circular tests, and oracle gaming, but does not provide implementation procedures for correcting them.
Examples
Uses flaky tests, circular tests, and agent shortcutting as examples of misleading validation.
What you can do after
Diagnose misleading test feedback before optimizing CI speed or expanding automated development.
How it connects
Connects test reliability to CI speed, agent iteration, automated review, and continuous modernization.

Also explained in

Compare how 21 books explain reliability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.