liveBook · Manning

AI-generated pull request

As explained in Vibe Engineering

Source code produced in whole or in part by an artificial-intelligence system.

An AI-generated pull request proposes changes containing code produced wholly or partly by an AI system. Generation can make prototypes, migrations, and tests cheaper to produce, but a plausible diff is not evidence that the change is correct, safe, or understood. The work moves downstream to verification and ownership. section 1.0 section 2.3.2

A green test suite misses a scaling failure

A generated change fetches related records one at a time inside a loop. It passes tests and works with 50 staging rows, but times out when a customer has 40,000 rows. The on-call engineer must first reconstruct the code’s intent before fixing it. This is trust debt: an immediate saving with a deferred, shared cost. section 1.4

Compilation, type checking, and passing examples are signals, not proof of correctness. Tests are most useful when they express the intended behavior as a contract, rather than merely confirm what the generated implementation happens to do. Review must also ask whether the code fits its architecture and business intent. section 8.2.1 section 8.3 section 1.8

A disciplined path to merge is:

  • Specify behavior and constraints before generation; turn important requirements into testable contracts. section 1.8
  • Review the diff for architectural fit, edge cases, security, and performance; use property, mutation, performance, and threat-model-derived tests where appropriate. section 1.6 section 2.3.2
  • Separate code verification from control over agent actions. Use boundaries such as sandboxing, dry runs, and least-privilege credentials for actions that tests alone cannot make safe. section 1.1.4
  • Roll out cautiously with metrics that check the risks identified in review; use a smaller initial canary when staging provides limited confidence. section 8.7.3
Table 1.1 Examples of AI Failure Classes
Case
Failure class
What was missing
Where the fix lives
Enrichlead
Insecure generated code
Input validation, rate limiting, auth - and the adversarial tests that would have exposed their absence
Specification and verification. Testable, and the rest of this book shows how.
Gemini CLI file loss
Unverified action on the world
A check that file-system state matched what the agent believed
Architecture and permissions. No test suite helps; the agent needed a sandbox and a dry run.
NX supply chain
Trusted input, untrusted content
Human review of an AI-authored PR touching the build system
Process. A review gate with authority to block, applied by change risk rather than change size.
Replit data deletion
Unconstrained autonomy
A boundary the agent could not cross regardless of what it concluded
Governance. Least-privilege credentials, environment separation, irreversible actions behind human approval.

Compare failures that call for different controls: adversarial tests, action safeguards, review gates, and autonomy boundaries. section 1.1.4

Human review should be meaningful and recorded where governance requires it; repeated permission clicks are not a substitute. Audit trails, compliance expectations, and ownership or licensing questions may need explicit decisions before deployment. section 5.5

How Vibe Engineering covers it

How deep it goes
Develops a verification workflow with contracts, adversarial tests, review, permission boundaries, and staged rollout, while identifying unresolved governance questions.
Examples
Illustrates a loop of database fetches that passes staging tests but times out at customer scale, creating trust debt.
What you can do after
Evaluate and understand generated changes using explicit contracts, risk-based review, and verification before merging.
How it connects
Connects generated changes to executable specifications, code review, automation bias, security testing, least privilege, and progressive rollout.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.