An AI-generated pull request proposes changes containing code produced wholly or partly by an AI system. Generation can make prototypes, migrations, and tests cheaper to produce, but a plausible diff is not evidence that the change is correct, safe, or understood. The work moves downstream to verification and ownership. section 1.0 section 2.3.2
A green test suite misses a scaling failure
A generated change fetches related records one at a time inside a loop. It passes tests and works with 50 staging rows, but times out when a customer has 40,000 rows. The on-call engineer must first reconstruct the code’s intent before fixing it. This is trust debt: an immediate saving with a deferred, shared cost. section 1.4
Compilation, type checking, and passing examples are signals, not proof of correctness. Tests are most useful when they express the intended behavior as a contract, rather than merely confirm what the generated implementation happens to do. Review must also ask whether the code fits its architecture and business intent. section 8.2.1 section 8.3 section 1.8
A disciplined path to merge is:
- Specify behavior and constraints before generation; turn important requirements into testable contracts. section 1.8
- Review the diff for architectural fit, edge cases, security, and performance; use property, mutation, performance, and threat-model-derived tests where appropriate. section 1.6 section 2.3.2
- Separate code verification from control over agent actions. Use boundaries such as sandboxing, dry runs, and least-privilege credentials for actions that tests alone cannot make safe. section 1.1.4
- Roll out cautiously with metrics that check the risks identified in review; use a smaller initial canary when staging provides limited confidence. section 8.7.3
Table 1.1 Examples of AI Failure Classes
|
Case
|
Failure class
|
What was missing
|
Where the fix lives
|
|
Enrichlead
|
Insecure generated code
|
Input validation, rate limiting, auth - and the
adversarial tests that would have exposed their absence
|
Specification and verification. Testable,
and the rest of this book shows how.
|
|
Gemini CLI file loss
|
Unverified action on the world
|
A check that file-system state matched what the agent
believed
|
Architecture and permissions. No
test suite helps; the agent needed a sandbox and a dry run.
|
|
NX supply chain
|
Trusted input, untrusted content
|
Human review of an AI-authored PR touching the build
system
|
Process. A review gate with
authority to block, applied by change risk rather than change size.
|
|
Replit data deletion
|
Unconstrained autonomy
|
A boundary the agent could not cross regardless of what it
concluded
|
Governance. Least-privilege
credentials, environment separation, irreversible actions behind human
approval.
|
Compare failures that call for different controls: adversarial tests, action safeguards, review gates, and autonomy boundaries. section 1.1.4
Human review should be meaningful and recorded where governance requires it; repeated permission clicks are not a substitute. Audit trails, compliance expectations, and ownership or licensing questions may need explicit decisions before deployment. section 5.5
