Acceptance rate measures the share of AI-generated suggestions or outputs that developers accept. In model comparisons, acceptance means using output without significant modification. section 9.2.4
Treat it as a workflow-specific quality signal, not a universal score: measured rates vary across teams, tasks, prompts, and codebases. Lower rates can also mean more retries or manual corrections. section 9.1.3 section 9.2.4
Use the metric to guide engineering decisions:
- Compare models by task: trivial suggestions showed little differentiation, while complex debugging and multi-file work showed larger gaps. section 9.2.4
- Check context practices: providing relevant files and clear instructions was associated with higher acceptance than supplying broad, tangential context. section 9.2.3
- Include human rework in cost decisions; cheaper output can cost more overall when developers retry or override it more often. section 9.1.3
- Measure local model changes on representative tasks: one 500-completion test found less than a 2% difference between two quantization levels. section 9.3.5
Table 9.1 Change of costs between models
|
Task Type
|
Sonnet
|
Haiku
|
GPT-4.1-mini
|
Model to be selected
|
|
Tab completion
|
94%
|
93%
|
92%
|
Any
(≈)
|
|
Import statements
|
97%
|
96%
|
96%
|
Any
(≈)
|
|
Boilerplate generation
|
91%
|
88%
|
87%
|
Any
(≈)
|
|
Unit test scaffolding
|
89%
|
82%
|
80%
|
Sonnet*
|
|
Inline refactoring
|
90%
|
83%
|
81%
|
Sonnet*
|
|
Function implementation
|
88%
|
74%
|
72%
|
Sonnet
|
|
Complex debugging
|
85%
|
61%
|
58%
|
Sonnet
|
|
Multi-file refactoring
|
82%
|
52%
|
49%
|
Sonnet
|
|
Architecture planning
|
79%
|
41%
|
38%
|
Sonnet
|
Compare reported acceptance rates across model tiers and coding task types. section 9.2.4
