liveBook · Manning

Quality sampling

As explained in Vibe Engineering

Quality sampling is the periodic manual review of a random subset of output changes for qualitative defects that automated checks may miss.

Quality sampling is a recurring manual check against silent quality erosion: review a random subset of merged AI-generated changes for defects that automated checks do not measure. section 9.4.3

A practical sampling loop:

  • Review ten randomly selected AI-generated changes each week, focusing on naming, architectural alignment, unnecessary complexity, and missed test edge cases rather than bugs already caught by CI. section 9.4.3
  • Track recurring findings alongside first-attempt success, developer feedback, and other health signals; sampling supplies evidence that aggregate metrics may miss. section 9.4.3
  • Reduce sampling frequency only after stable results justify lower monitoring overhead; one team moved from weekly to biweekly after three stable months. section 9.4.4

A regression automated metrics missed

A model update changed TypeScript generic handling, producing technically correct but unnecessarily complex annotations. Sampling exposed the accumulated complexity even though CI reported no bugs. section 9.4.3

Figure 9.7 Visualization of the potential dashboard - state of the art.

Use the dashboard visualization to situate quality sampling among the broader health-check signals and monitoring decisions. section 9.4.4

How Vibe Engineering covers it

How deep it goes
Defines a weekly ten-change review with explicit criteria, then discusses reducing cadence after stable metrics.
Examples
Uses random ten-change reviews and a TypeScript generics regression whose complexity escaped automated bug detection.
What it assumes
Assumes automated checks already catch bugs and adds human review for naming, architecture, complexity, and test coverage quality.
What you can do after
Diagnose quality regressions by reviewing representative merged changes and adjusting monitoring cadence as confidence grows.
How it connects
Connects quality sampling with continuous integration, first-attempt success rate, developer feedback, and health-check monitoring.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.