liveBook · Manning

Progressive evaluation

As explained in Vibe Engineering

Progressive validation is a staged verification approach that applies increasingly costly checks only when earlier checks pass or indicate they are needed.

Running every test after every change may seem prudent after a regression, but at large scale it can make feedback too slow. Progressive validation stages checks by cost and relevance, escalating only as needed. section 10.3.4

Use three tiers:

  • Run local linters and static checks first to catch trivial failures quickly and keep them out of shared CI queues. section 10.3.4
  • Select CI tests based on changed files and their dependencies: a checkout change runs checkout tests and direct consumers’ tests, while a leaf utility may need only 12 tests. section 10.3.4
  • If selected tests still fail, send the failure to the agent; cap repair attempts at two, then flag a human rather than spending more compute. section 10.3.4

The sequence depends on trustworthy tests: fix flaky or misleading tests before optimizing feedback speed. A fast, unreliable signal can send agents chasing false failures or create false confidence. section 10.9

How Vibe Engineering covers it

How deep it goes
Defines three tiers—local checks, selective CI, and a two-attempt repair cap—without detailing their implementation.
Examples
Uses Stripe’s three-tier loop, including checkout tests for checkout changes and 12 tests for a leaf utility.
What it assumes
Relies on reliable tests and dependency tracking; flaky tests undermine feedback, while dependency graphs help select relevant checks.
What you can do after
Apply staged verification with local checks, dependency-selected CI tests, and a human escalation after two failed repair attempts.
How it connects
Connects linters, selective testing, dependency graphs, continuous integration, and test reliability.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.