liveBook · Manning

First-attempt success rate

As explained in Vibe Engineering

First-attempt success is the rate at which an AI-assisted task is completed successfully without requiring a retry or correction.

A coding assistant can appear fast while repeated corrections consume developer time. First-attempt success rate measures the percentage of AI-generated code that passes code review without modification. It is a quality metric with a direct bearing on retry costs, not a complete measure of AI return on investment. section 9.4.2

Fewer corrections after optimization

A team measured 72% first-attempt success before optimization and 81% after improving prompts and selecting models by task type. The nine-percentage-point increase meant fewer retries and less time correcting output. The rate should still be considered alongside cycle time, throughput, test coverage, and developer satisfaction; no single metric establishes ROI. section 9.4.2

Track the rate over time rather than assuming output quality is stable. Small declines can be hard to notice in individual suggestions but add up to more review work and less trust. A hypothetical drop from 81% to 73% over eight weeks illustrates how quality erosion can remain hidden while the tool stays in use. section 9.4.3

Make the metric actionable:

  • Sample output quality regularly. One team lead reviews ten randomly selected merged AI-generated changes each Friday for issues such as naming and architectural alignment. section 9.4.3
  • Set an investigation trigger. In one health check, a rate below 78% prompts a review of model and prompt changes from the previous two weeks; the threshold is a team-specific playbook example. section 9.4.4
  • Treat repeated poor suggestions as a possible configuration signal, and consider switching models rather than quietly working around degradation. section 9.4.3
  • Reduce monitoring overhead when results are stable. One team shifted quality sampling from weekly to biweekly after three stable months. section 9.4.4

How Vibe Engineering covers it

How deep it goes
Defines review-based success and outlines sampling, trend monitoring, and an investigation trigger.
Examples
Reports a team's improvement from 72% to 81% and models a subsequent decline from 81% to 73%.
What you can do after
Evaluate AI coding quality over time and investigate model or prompt changes when the rate falls below a chosen threshold.
How it connects
Connects code review, quality erosion, model selection, prompt optimization, and AI return on investment.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.