Chapter 8: Evaluation is king: how to trust code you didn’t write
The introduction explains why AI-generated code creates a distinct trust problem and calls for confidence based on verified properties rather than an AI’s assurance.
After this chapter you will be able to
- AI Code Trust Deficit. Explain why the section likens newly generated AI code to legacy code, even when it has high test coverage.
- Tests Without Author Intent. Distinguish the role of testing human-written code from its role in evaluating AI-generated code, and explain why reviewing AI like a junior developer is insufficient.
- Evidence Based Confidence. Describe how a person guiding AI-generated code should express confidence in it for a particular use case.
- Generation and evaluation asymmetry. Explain why producing code with an AI assistant is easier than deciding whether that code is trustworthy enough to use.
- Generation and evaluation bottleneck. Explain why AI-generated code shifts the bottleneck from producing code to evaluating it, and why engineering trust becomes a central development skill.
- Trust deficit and false shortcuts. Describe the trust deficit and explain why reviewing faster or simply trusting AI does not resolve it.
- Polished code hides errors. Explain why bad AI-generated code can evade the review instincts that catch obviously bad human-written code.
- Multiplying review burden. Explain why generating substantially more plausible-looking code can increase potential debt faster than code volume alone suggests.
- Changed meaning of works for me. Distinguish the older environment-reproduction problem from what “works for me” often means for AI-generated code: only that a few tried inputs produced plausible-looking outputs.
- Limits of passing tests. Explain why a passing case gives little reason to expect AI-generated code to handle neighboring cases, and state the asymmetry between finding a bug and establishing that none exist.
- SWE-bench score validity. Explain why SWE-bench is closer to real development than HumanEval, yet a successful patch does not necessarily demonstrate a sound fix.
- Missing real-world engineering work. Describe how SWE-bench's issue selection and solo patch format leave out difficult parts of real software engineering.
- Benchmark Scores Versus Production Readiness. Explain why a high benchmark score is not evidence that code can ship without review, and describe the role of evaluation in closing that gap.
- Trust Across Evaluation Gates. Trace how code accumulates confidence through local development, pull request review, CI, staging, and production, describing the different failures each gate can reveal.
- Property Checks for AI Code. Explain why existing evaluation gates can be less reliable for AI-generated code and why they must verify properties rather than prompt an author's memory.
- Plausible Parser Hidden Failures. Explain why the example AI-generated markup parser can look ready to ship while still being broken, and contrast its straightforward cases with the failures described here.
- Always On Verification Baseline. Explain why verification gates should run on every piece of code regardless of origin, and state what those gates can and cannot establish.
- Types versus intended behavior. Distinguish what a compiler establishes about a function from whether that function performs its intended task.
- AI and the pesticide paradox. Explain why type checking may be especially unlikely to expose the errors in AI-generated code, using the section’s pesticide-paradox analogy.
- Static analysis and its reach. Explain how linters and rule engines examine code patterns beyond syntax and types, and why this makes them useful for checking large amounts of AI-generated code.
- Domain bugs beyond static rules. Explain why passing static checks does not show that the parser handles markup correctly, using the four-hash header bug as an example.
- Useful review and missing context. Explain how AI review can identify potential code issues while failing to judge whether behavior matches a project's requirements or conventions.
- Targeted checks and independent review. Describe how explicit checklists, non-LLM hints, and a different review model improve AI checks, and why human review remains necessary.
- Risks of AI-selected dependencies. Explain why an AI-suggested library needs scrutiny rather than being treated as a deliberate, vetted choice. Describe the risks of unsuitable, vulnerable, incompatible, unnecessary, or nonexistent packages, including slopsquatting.
- Dependency verification checks. Describe the automated checks and human review recommended for AI-generated dependencies, including checking necessity and confirming a package exists before installation.
- Requested change boundaries. Explain how diff-based checks compare the size and location of an AI-generated change with the original request, and why unrelated edits deserve scrutiny.
- Behavior behind small diffs. Describe why semantic diff analysis checks behavior and interfaces rather than relying only on line counts.
- Static Gates and Behavioral Confidence. Distinguish what always-on gates establish about the parser from what they cannot establish, and explain why checking its behavior requires executing it with tests.
- Tests as behavioral specifications. Explain why tests take on a specification role for AI-generated code: they state the intended behavior that a developer can inspect, while the generated code is one possible implementation.
- Unexpected behavior through exploration. Explain why manual exploration can reveal problems that automated tests miss, especially when the expected behavior has not yet been specified.
- Manual reconnaissance before automation. Describe how to use a short manual exploration of significant AI-generated code to decide what to automate, without treating exploration as the entire verification strategy.
- Shared Edge Case Blind Spots. Explain why unit tests remain useful yet can give false confidence about AI-generated code, especially on edge cases and combinations.
- Tests as AI specifications. Explain how test-driven prompting uses tests in the prompt to specify the behavior AI-generated code must produce, including edge cases.
- Specification-driven generation workflow. Trace the section's workflow from writing behavioral tests through generating code, running tests, and adding tests for newly discovered cases; distinguish it from traditional one-test-at-a-time TDD.
- Limits of anticipated test cases. Explain why test-driven prompting cannot protect against behavior the developer has not thought to test, using the nested-formatting example.
- Coverage versus behavioral verification. Distinguish code coverage from evidence that tests verify correct behavior. Explain why inputs that execute the same lines can still reveal bugs the tests miss.
- Shared AI test blind spots. Explain why asking an AI to generate both an implementation and its tests can make high coverage especially misleading. Describe how tests that mirror the implementation can miss edge cases or repeat its logic.
- Golden output regression checks. Describe how a snapshot or golden test saves a program's output and compares later runs against it to detect changes.
- Incorrect golden baselines. Explain why a passing snapshot test does not prove that AI-generated output is correct, and when snapshots are useful.
- Human-approved golden outputs. Explain how approval testing differs from basic snapshot testing: a person reviews an output and explicitly approves it before it becomes a golden standard.
- Visual regression review. Describe how visual regression testing uses screenshots of UI states, approved baselines, and subsequent visual diffs to identify changes that a human can review.
- Quality versus visual stability. Distinguish visual regression against an approved baseline from reference-based evaluation against an external example, and explain why a team may use both at different stages.
- Tests as behavioral contracts. Explain how tests can state what code must do without prescribing how it works, and why passing them is evidence of meeting a contract rather than proof of correctness.
- Example-Based Testing Blind Spots. Explain why passing unit tests does not show that AI-generated code handles inputs the developer never thought to test.
- Adversarial Verification Approach. Describe how adversarial verification seeks failures beyond the examples a developer chooses.
- Properties Instead of Examples. Explain how property-based testing differs from example-based testing: the author states a rule that should hold across inputs, and a framework generates inputs to challenge it.
- Parser Properties Worth Checking. Describe the parser properties used as examples: identical inputs give identical outputs, plain text without markup passes through unchanged, and arbitrary inputs should produce a string rather than crash. Distinguish these from the proposed well-formed-HTML property.
- Uninformative Passing Properties. Explain why the passing plain-text property offers limited confidence: excluding every markup character restricts the generated inputs to ones the parser ignores.
- Random Generation Blind Spots. Explain why the no-crash property passed even against the vulnerable regex: the failure needs a long, monotonous input, while random generation favors shorter, varied strings. State the section's distinction between that approach and fuzzing that mutates toward unexplored paths.
- Input Independent Invariants. Explain that an invariant should hold for every input, and describe balanced output tags as a useful invariant for the parser.
- Conditional Metamorphic Relations. Describe how a metamorphic relation connects outputs when inputs are changed, and explain why the parser's concatenation relation fails unless sections are independent.
- Conserved Link Counts. Explain conservation laws as properties that preserve a quantity, using the parser's expected one-to-one correspondence between markdown links and HTML links.
- Property Failures Reveal Specifications. Explain why a failing property test can reveal an unresolved definition rather than a faulty parser, using the disagreement over whether []() is a link.
- Coverage-guided fuzzing loop. Explain how coverage-guided fuzzing uses feedback from instrumented code to explore inputs likely to reach new branches, rather than relying on random inputs alone.
- LLM-guided failure inputs. Describe how LLM-guided fuzzing proposes plausible but tricky inputs and why its objective differs from that of an LLM generating code.
- Mutation survival and test quality. Explain how mutation testing changes code, reruns the tests, and uses surviving mutations to reveal behaviors the tests do not verify. Distinguish mutation score from code coverage using the parser example.
- Syntactic and semantic mutations. Explain why conventional small syntactic mutations may fail to represent faults in AI-generated code, and describe the kind of behavior-level mutation an LLM can generate instead.
- Disagreement as a bug signal. Explain how differential testing compares independent implementations of the same specification on generated inputs, and why a disagreement calls for investigation rather than automatically identifying the correct output.
- Adversarial verification mindset. Explain how adversarial verification differs from checking expected cases, and why the section says it is especially necessary for AI-generated code.
- Failures beyond the component. Explain why a correctly tested parser can still cause problems in a real system, and what system-level verification examines instead.
- Explicit function contracts. Explain how Design by Contract turns a function’s implicit assumptions and promises into preconditions, postconditions, and invariants, especially when AI-generated code has no human author holding those assumptions in mind.
- Executable boundary checks. Describe how runtime contract checks expose integration errors at a function boundary instead of letting failures propagate into less obvious behavior.
- Consumer and provider contracts. Explain how contract testing checks an API relationship between services: the consumer declares what it will send and expects back, and the provider verifies that it delivers that response.
- AI-generated API breakage. Describe why an AI-generated parser change can break another service despite appearing to improve the parser, and how a contract test exposes that break.
- Independent consumer-owned verification. Explain why Pact-style contracts live with the consumer and are verified independently of provider changes, and state the participation and overhead this requires.
- Integration and E2E test scope. Distinguish what unit, contract, integration, and end-to-end tests verify. Trace how an integration test checks the parser with surrounding systems, while an E2E test checks the entire user journey.
- Behavior-focused test resilience. Explain why integration and E2E tests are valuable when AI refactors or replaces an implementation: they check the user's observable outcome rather than how the parser works internally.
- Functional failures as review signals. Explain why self-healing a functional test can conceal an AI-generated regression, and why a behavior-changing failure should prompt human review instead of automatic adaptation.
- Unseen behavioral regressions. Explain how an AI change can pass tests for its requested feature yet alter existing behavior the AI did not realize mattered.
- Characterization tests for preservation. Describe how to capture behavior before a change and compare it afterward, allowing only intended differences.
- Call graph impact analysis. Trace how impact analysis identifies callers of changed functions, determines the behaviors they rely on, and targets tests to those dependencies.
- Proofs versus sampled tests. Distinguish testing's checks of selected inputs from formal methods' aim of proving correctness for all cases covered by a specification.
- Specification and intent gap. Explain why proving code correct against a specification does not prove that the specification expresses what people intended, including when AI drafts it.
- Refinement Types as Compile-Time Guarantees. Explain how refinement types add constraints to ordinary types and let the compiler reject code that could violate them, including AI-generated code that might otherwise require a test to catch.
- Constraint-based code verification. Explain how a SAT or SMT solver uses logical constraints to check a code property, and what “sat” and “unsat” mean for the property being checked.
- LLM-Guided Formal Exploration. Describe how an LLM guides symbolic execution toward likely bug-revealing paths, what each technique contributes, and the approach's current maturity.
- Formal proof and verification tools. Distinguish formal verification from testing and type checking, and describe what TLA+ and Dafny verify in this section's examples.
- AI-assisted verification responsibilities. Describe how AI assists formal verification, what the verifier and human must still do, and where the section recommends using this approach.
- Formal verification decision criteria. Explain why formal methods are overkill for most code and describe the circumstances that can justify them: catastrophic bugs, naturally formal domains, persistent subtle corner cases, or regulatory demands.
- Limits of pre-deployment testing. Explain why even extensive testing cannot rule out production surprises: tests rely on simplified models of users, data, load, and environments.
- Deliberate runtime verification. Describe runtime verification as using production deliberately to detect, contain, and learn from failures that could not be fully verified beforehand.
- Unpredictable AI change impact. Explain why even a small request to an AI coding tool warrants regression testing in CI: the resulting code change may be much broader than requested, making its blast radius hard to predict.
- Regression test scope tradeoff. Describe the two CI testing approaches presented: tag tests by component and dependency to run potentially affected tests, or run the entire suite for every AI change. Explain the extra CI time versus the chance of catching unexpected regressions.
- Production logs as test inputs. Explain how logging unusual, slow, or error-triggering parser inputs reveals cases missed by testing and supplies inputs for further investigation and tests.
- Request tracing for diagnosis. Describe how tracing follows a request through services, parser, and renderer to show where an incorrect result arose, even when AI-generated code is hard to reason about.
- Metrics as production signals. Explain how aggregate production metrics reveal system health and changes after deploying AI-generated code, making real usage an ongoing source of evaluation.
- Cautious rollout of generated code. Explain why staged rollout carries more weight for generated code than for human-written code, and describe how the rollout should change.
- Production comparison through A/B testing. Distinguish the question A/B testing answers from the correctness question answered by other evaluation methods in the chapter.
- Safe failure under stress. Explain how chaos engineering introduces failures to check whether AI-generated code degrades gracefully rather than corrupting state, hanging, or crashing.
- Non-functional quality checks. Explain how non-functional tests assess qualities that functional tests can miss, and describe what performance, security, accessibility, and compatibility tests check in AI-generated code.
- Selective verification choices. Explain why evaluation requires choosing which verification techniques to apply where, rather than using every available technique on every project.
- Risk-calibrated confidence in code. Explain how the cost of incorrect code determines the confidence needed and the evaluation effort worth investing. Illustrate the difference using the parser in a personal blog, a news organization, and a browser-facing application.
- Risk-based verification investment. Explain why verification effort should rise with system criticality: minimal effort for low-cost failures, standard verification for most production features, and intensive verification when failures could be severe.
- Verification methods by risk. Apply the section's method choices: use cheap always-on gates for all code, adversarial checks for untrusted input or complex logic, system-level checks for integrations, and formal methods or runtime verification for critical code.
- Debug regenerate or trust tests. Apply the section's decision framework to choose whether to debug, regenerate, step back, or rely on test evidence.
- Minimum Viable Verification Cadence. Describe the four verification cadences for an AI-first team and the checks assigned to each: every change, before merge, regularly, and before major releases.
- Human Evaluation Competence. Describe the judgment needed to evaluate AI-generated code, including understanding its behavior, questioning the tests, and being candid about remaining uncertainty.
- QA Skills Become Central. Explain why automated code generation raises the value of systematic testing, adversarial thinking, edge-case intuition, and failure-mode analysis.
- Trust Engineering Mindset. Describe the shift from producing code to deciding whether code deserves trust, including the move from expected behavior to probing failure paths and judging their risks.
What it builds on
- Trust: a new kind of debt (chapter 1) · The earlier discussion of trust as a kind of debt.
Section by section
Introduction
The introduction explains why AI-generated code creates a distinct trust problem and calls for confidence based on verified properties rather than an AI’s assurance.
- AI Code Trust Deficit: Explain why the section likens newly generated AI code to legacy code, even when it has high test coverage.
Why it matters: A coverage number does not, by itself, resolve uncertainty about whether AI-generated code does the right thing. - Tests Without Author Intent: Distinguish the role of testing human-written code from its role in evaluating AI-generated code, and explain why reviewing AI like a junior developer is insufficient.
Why it matters: The usual review conversation cannot substitute for evidence when there is no author whose intent can be checked. - Evidence Based Confidence: Describe how a person guiding AI-generated code should express confidence in it for a particular use case.
Why it matters: Specific evidence makes uncertainty discussable when deciding whether code is fit for its intended use.
Key terms: legacy code, code coverage, trust deficit, author's intent, mental model, code review, verified properties, confidence level, use case
8.1 Evaluation matters more than generation
This section shifts attention from how quickly AI generates code to whether the generated code can be trusted in real use.
- Generation and evaluation asymmetry: Explain why producing code with an AI assistant is easier than deciding whether that code is trustworthy enough to use.
Why it matters: Fast generation is valuable only if you can decide whether the result is fit to use.
Key terms: generation, evaluation
8.1.1 Code is now produced faster than confidence
AI accelerates code generation without increasing thoughtful review capacity, creating a trust deficit that cannot be solved by rushing review or blindly trusting the AI.
- Generation and evaluation bottleneck: Explain why AI-generated code shifts the bottleneck from producing code to evaluating it, and why engineering trust becomes a central development skill.
Why it matters: Generating code faster is useful only if the team can still establish confidence in what it ships. - Trust deficit and false shortcuts: Describe the trust deficit and explain why reviewing faster or simply trusting AI does not resolve it.
Why it matters: Neither a quicker sign-off nor blind confidence in AI makes unverified code safer.
Key terms: engineering trust, generation, evaluation, trust deficit, adequate verification, production incident queue
8.1.2 Scale of generation creates scale of potential debt
The section explains how AI-generated code can multiply hidden trust debt because its polished appearance makes mistakes harder to spot as code volume grows.
- Cognitive and trust debt: Distinguish the cost to a team of rebuilding missing understanding from the cost to users when code ships without anyone truly understanding it.
Why it matters: These terms name who bears the cost when generated code is shipped without understanding. - Polished code hides errors: Explain why bad AI-generated code can evade the review instincts that catch obviously bad human-written code.
Why it matters: A professional-looking diff can still contain errors that ordinary visual review misses. - Multiplying review burden: Explain why generating substantially more plausible-looking code can increase potential debt faster than code volume alone suggests.
Why it matters: Faster generation does not mean a team can build confidence at the same rate.
Key terms: trust debt, cognitive debt, plausible-looking code, invariants, verified logic, scale of generation
8.1.3 "Works for me" means nothing now
The section explains why a few successful checks provide much less confidence in AI-generated code than “works on my machine” once did.
- Changed meaning of works for me: Distinguish the older environment-reproduction problem from what “works for me” often means for AI-generated code: only that a few tried inputs produced plausible-looking outputs.
Why it matters: A convincing result on your chosen inputs is not enough to trust the generated code. - Limits of passing tests: Explain why a passing case gives little reason to expect AI-generated code to handle neighboring cases, and state the asymmetry between finding a bug and establishing that none exist.
Why it matters: The section's reason for demanding stronger evaluation is that passing examples cannot supply the missing confidence about related behavior.
Key terms: works on my machine, works for me, AI-generated code, mental model, inductive power, counterexample
8.1.4 How the community measures "good AI code" (and why it's misleading)
This section explains why coding benchmark scores can show progress on narrow tasks without establishing that AI-generated code is trustworthy in production.
- Isolated coding benchmark limits: Explain what HumanEval measures, how EvalPlus improves its tests, and why neither reflects the context of production development.
Why it matters: A better score on small programming problems does not by itself show that a model can make a safe change to your system. - SWE-bench score validity: Explain why SWE-bench is closer to real development than HumanEval, yet a successful patch does not necessarily demonstrate a sound fix.
Why it matters: Even a benchmark built from real repositories can count a patch as successful for reasons other than a correct, independently found solution. - Missing real-world engineering work: Describe how SWE-bench's issue selection and solo patch format leave out difficult parts of real software engineering.
Why it matters: A model's ability to produce a test-passing patch does not measure how it handles unclear requirements or works with a team.
Key terms: HumanEval, EvalPlus, codebase context, SWE-bench, solution leakage, inadequate test cases, clear test, collaborative
8.1.5 The benchmark-reality gap
Explains why strong coding benchmark scores do not establish that code is ready for production, and why evaluation must bridge that gap.
- Benchmark Scope and Limits: Distinguish the coding abilities these benchmarks measure from the production qualities they do not measure.
Why it matters: A benchmark score is useful only if you know which qualities it does and does not test. - Benchmark Scores Versus Production Readiness: Explain why a high benchmark score is not evidence that code can ship without review, and describe the role of evaluation in closing that gap.
Why it matters: It keeps a leaderboard result from being mistaken for a decision to ship.
Key terms: HumanEval, MBPP, SWE-bench, domain, adversarial input, maintain, benchmark-reality gap, production readiness, evaluation, clear boundaries
8.1.6 Where evaluation actually lives
This section explains how code gains trust through successive evaluation gates and why AI-generated code requires gates that verify properties rather than rely on an author's judgment.
- Trust Across Evaluation Gates: Trace how code accumulates confidence through local development, pull request review, CI, staging, and production, describing the different failures each gate can reveal.
Why it matters: A passing check tells you what has been checked, not that the code is fully trustworthy. - Early Detection Cost Tradeoff: Explain why an evaluation pipeline should catch as many failures as possible at lower-cost gates before they reach production.
Why it matters: Finding a defect before it reaches users avoids the cost of discovering it in production. - Property Checks for AI Code: Explain why existing evaluation gates can be less reliable for AI-generated code and why they must verify properties rather than prompt an author's memory.
Why it matters: A plausible-looking answer from an AI is not evidence that the generated code is safe.
Key terms: evaluation pipeline, gates, local development, pull request review, CI pipeline, staging environment, lowest cost, user impact, AI-generated code, human failure modes
8.2 Always-on verification
Introduces a deceptively plausible AI-generated markup parser and explains why cheap, always-on verification gates are a necessary baseline, though not proof of correctness.
- Plausible Parser Hidden Failures: Explain why the example AI-generated markup parser can look ready to ship while still being broken, and contrast its straightforward cases with the failures described here.
Why it matters: Code that passes obvious examples can still fail on inputs a user will encounter. - Always On Verification Baseline: Explain why verification gates should run on every piece of code regardless of origin, and state what those gates can and cannot establish.
Why it matters: A fast check is valuable as a routine first filter, not as a reason to trust the code completely.
Key terms: lightweight markup language, happy path, nested formatting, always-on verification, gates, baseline
8.2.1 Compilation and type checking: a signal, not proof
Compilation and type checking catch structural errors, but type-correct AI-generated code can still be logically wrong.
- Language-dependent verification baseline: Explain how language choice affects what compilation and type checking can catch in the markup-parser example.
Why it matters: The language determines how much reassurance this first verification gate can provide. - Types versus intended behavior: Distinguish what a compiler establishes about a function from whether that function performs its intended task.
Why it matters: A successful type check is useful evidence, but it is not proof that the code does the right thing. - AI and the pesticide paradox: Explain why type checking may be especially unlikely to expose the errors in AI-generated code, using the section’s pesticide-paradox analogy.
Why it matters: AI code can pass familiar gates on its first commit while still failing the behavior you need.
Key terms: dynamic typing, TypeScript, type checking, verification baseline, string -> string, correctness, pesticide paradox, AI-generated code, logically wrong
8.2.2 Static analysis: linters, rule engines, and SAST
The section shows how static analysis can quickly flag risky code patterns in an AI-generated parser, while explaining why it cannot establish that the parser behaves correctly.
- Static analysis and its reach: Explain how linters and rule engines examine code patterns beyond syntax and types, and why this makes them useful for checking large amounts of AI-generated code.
Why it matters: Fast, repeatable checks help keep the volume of generated code from overwhelming reviewers. - Regex risk and rule trade-offs: Describe why the nested-quantifier regex in the parser example can cause ReDoS, why the original delimiter-based regex is different, and what trade-off the security rule makes.
Why it matters: The example shows both the value of a targeted rule and the need to examine its warnings rather than assume every regex is unsafe. - Potential XSS versus confirmed vulnerability: Explain why a SAST scanner might flag the parser's unescaped HTML output without being able to confirm an exploitable vulnerability.
Why it matters: A scanner finding calls for investigation, not an automatic conclusion that the application is exploitable. - Domain bugs beyond static rules: Explain why passing static checks does not show that the parser handles markup correctly, using the four-hash header bug as an example.
Why it matters: Clean automated checks leave a separate question: does the code do the right thing for its domain?
Key terms: static analysis, linters, rule engines, code smells, ReDoS, nested quantifier, backtracking, eslint-plugin-security, Static Application Security Testing, SAST
8.2.3 AI code review: rules, checklists, and heuristics
AI code review can catch useful issues, but it works best as a targeted check alongside independent tools and human judgment.
- Useful review and missing context: Explain how AI review can identify potential code issues while failing to judge whether behavior matches a project's requirements or conventions.
Why it matters: A plausible finding still needs to be checked against what the software is meant to do. - Correlated reviewer blind spots: Explain why an LLM reviewer is not fully independent of generated output and how that can leave recurring errors undetected.
Why it matters: An AI approval can share the generator's blind spots rather than provide independent reassurance. - Targeted checks and independent review: Describe how explicit checklists, non-LLM hints, and a different review model improve AI checks, and why human review remains necessary.
Why it matters: Concrete checks make AI review more useful without mistaking its approval for sign-off.
Key terms: AI code review, requirements, architectural conventions, LLM evaluators, perplexity, LLM-as-a-Judge, correlated failures, explicit checklists, targeted prompts, non-LLM analytic checker
8.2.4 Dependency, license, and security scanning
This section explains why AI-selected dependencies need verification, which checks to apply, and what dependency scanning cannot establish.
- Risks of AI-selected dependencies: Explain why an AI-suggested library needs scrutiny rather than being treated as a deliberate, vetted choice. Describe the risks of unsuitable, vulnerable, incompatible, unnecessary, or nonexistent packages, including slopsquatting.
Why it matters: A plausible package suggestion can introduce risk before anyone reviews the generated code. - Dependency verification checks: Describe the automated checks and human review recommended for AI-generated dependencies, including checking necessity and confirming a package exists before installation.
Why it matters: Separate checks catch different risks, while questioning necessity can avoid adding an attack surface at all. - Limits of dependency scanning: Distinguish what a clean dependency scan establishes from what it cannot establish about AI-generated code.
Why it matters: A green dependency report should not be mistaken for approval of the generated code.
Key terms: dependency, hallucinated package names, Slopsquatting, vulnerability scanning, license scanning, supply chain verification, dependency necessity review, hallucination check, dependency scanning, all-green dependency scans
8.2.5 Diff-based sanity checks: what actually changed?
This section shows how to check whether an AI-generated change matches the requested size, scope, and behavior, while recognizing the limits of diff checks.
- Requested change boundaries: Explain how diff-based checks compare the size and location of an AI-generated change with the original request, and why unrelated edits deserve scrutiny.
Why it matters: A change you did not request may also be a change you did not think to test. - Behavior behind small diffs: Describe why semantic diff analysis checks behavior and interfaces rather than relying only on line counts.
Why it matters: A small-looking patch can still affect callers throughout a system. - Heuristic checks and limits: Explain how baseline expectations make diff checks cheap, programmable signals, and when those signals are less useful.
Why it matters: Treating diff rules as review prompts avoids mistaking a cheap warning for a verdict.
Key terms: diff-based analysis, size reasonableness, scope containment, scope creep, semantic diff analysis, function signatures, public interfaces, API contract, blast radius, baseline expectations
8.2.6 The limits of always-on gates
The section explains why always-on checks provide useful structural confidence but cannot establish that AI-generated code behaves correctly.
- Static Gates and Behavioral Confidence: Distinguish what always-on gates establish about the parser from what they cannot establish, and explain why checking its behavior requires executing it with tests.
Why it matters: Passing automated checks is useful, but it is not evidence that the parser produces the right output.
Key terms: always-on gates, static verification, type checking, tests
8.3 Tests as contracts, not as coverage
This section reframes tests for AI-generated code as inspectable definitions of correct behavior, rather than checks that code merely passes.
- Tests as behavioral specifications: Explain why tests take on a specification role for AI-generated code: they state the intended behavior that a developer can inspect, while the generated code is one possible implementation.
Why it matters: An explicit definition of correct behavior gives you something meaningful to check generated code against. - Passing tests and false confidence: Explain how AI-generated code can pass tests and have high coverage while still being broken if the tests specify the wrong behavior.
Why it matters: A green test suite is only reassuring if its tests describe the right behavior.
Key terms: tests as specification, tests as verification, intent, high coverage, tests that pass
8.3.1 Manual testing: still necessary, but different
Manual testing remains useful for AI-generated code as brief reconnaissance that reveals unexpected behavior and guides subsequent automated tests.
- Unexpected behavior through exploration: Explain why manual exploration can reveal problems that automated tests miss, especially when the expected behavior has not yet been specified.
Why it matters: Exploration helps you find cases you would not have known to put in a test yet. - Manual reconnaissance before automation: Describe how to use a short manual exploration of significant AI-generated code to decide what to automate, without treating exploration as the entire verification strategy.
Why it matters: A little exploration makes the tests you write for unfamiliar generated code more relevant.
Key terms: manual testing, automated test, human pattern recognition, reconnaissance, happy path, boundaries, automated testing
8.3.2 Unit tests: the foundation that can betray you
This section shows why passing unit tests can give false confidence in AI-generated code, using a parser bug to expose shared human and AI blind spots.
- Passing Tests and Parser Failure: Describe how the parser's initial passing tests failed to reveal a formatting bug, and what an additional test exposed.
Why it matters: A concrete failure shows why a green test suite is not the same as a correct parser. - Shared Edge Case Blind Spots: Explain why unit tests remain useful yet can give false confidence about AI-generated code, especially on edge cases and combinations.
Why it matters: It helps you treat passing unit tests as a foundation for confidence, not proof that overlooked inputs are safe.
Key terms: parser, unit tests, nested formatting, malformed, edge cases, false sense of verification, AI-generated code
8.3.3 TDD with AI and test-driven prompting
Shows how to give an AI executable tests as a coding specification, then verify its output while recognizing that the tests cover only cases the developer anticipated.
- Tests as AI specifications: Explain how test-driven prompting uses tests in the prompt to specify the behavior AI-generated code must produce, including edge cases.
Why it matters: The tests tell the AI what correct behavior looks like before it writes the code. - Specification-driven generation workflow: Trace the section's workflow from writing behavioral tests through generating code, running tests, and adding tests for newly discovered cases; distinguish it from traditional one-test-at-a-time TDD.
Why it matters: Generating against a test contract is useful only if you run the tests and update that contract as you learn. - Limits of anticipated test cases: Explain why test-driven prompting cannot protect against behavior the developer has not thought to test, using the nested-formatting example.
Why it matters: A passing test suite cannot reveal an important behavior that was never tested.
Key terms: Test-Driven Prompting, executable specifications, edge cases, Test-Driven Development, specification-driven generation, nested formatting, manual exploration
8.3.4 Code coverage: why AI can game it
The section explains why high code coverage can give false confidence in AI-generated code and tests.
- Coverage versus behavioral verification: Distinguish code coverage from evidence that tests verify correct behavior. Explain why inputs that execute the same lines can still reveal bugs the tests miss.
Why it matters: A high percentage can hide important untested behavior, even though coverage still helps identify code that never ran. - Shared AI test blind spots: Explain why asking an AI to generate both an implementation and its tests can make high coverage especially misleading. Describe how tests that mirror the implementation can miss edge cases or repeat its logic.
Why it matters: Tests are less reassuring when they repeat the assumptions behind the code they are meant to check.
Key terms: code coverage, lines, branches, or paths, correct behavior, tests and implementation are correlated, blind spots, adversarially
8.3.5 Snapshot and golden testing: where they help and where they rot
This section explains how snapshot tests detect changed output, why they cannot establish that the original output was correct, and why AI-generated code amplifies that risk.
- Golden output regression checks: Describe how a snapshot or golden test saves a program's output and compares later runs against it to detect changes.
Why it matters: A changed snapshot is a prompt to investigate, not a verdict about correctness. - Incorrect golden baselines: Explain why a passing snapshot test does not prove that AI-generated output is correct, and when snapshots are useful.
Why it matters: Otherwise, a green test suite can preserve a mistake rather than catch it. - AI amplified snapshot risk: Explain why the existing risk of approving a wrong snapshot arises earlier and at greater scale with AI-generated code.
Why it matters: AI generation makes it easier to bless many unexamined outputs on the first commit.
Key terms: snapshot testing, golden files, regression detection, golden standard, primary verification, regression, timing, volume, first snapshot
8.3.6 Approval testing and visual regression
This section explains how human-approved baselines verify outputs and UI changes, why visual stability is not the same as quality, and where review costs limit these methods.
- Human-approved golden outputs: Explain how approval testing differs from basic snapshot testing: a person reviews an output and explicitly approves it before it becomes a golden standard.
Why it matters: An approved baseline gives you more reason to trust a test than an output captured without review. - Visual regression review: Describe how visual regression testing uses screenshots of UI states, approved baselines, and subsequent visual diffs to identify changes that a human can review.
Why it matters: Screenshot comparisons can expose UI changes that a test for successful rendering would miss. - Quality versus visual stability: Distinguish visual regression against an approved baseline from reference-based evaluation against an external example, and explain why a team may use both at different stages.
Why it matters: A UI can remain perfectly unchanged while still being unsuitable to ship.
Key terms: approval testing, golden file, golden standards, visual regression testing, visual diffs, baseline, visual regression, reference-based evaluation, generate-evaluate-revise, regression baseline
8.3.7 Key shift: tests describe intent, not implementation
This section reframes tests for AI-generated code as statements of intended behavior, while explaining why passing tests cannot establish correctness beyond the cases tested.
- Tests as behavioral contracts: Explain how tests can state what code must do without prescribing how it works, and why passing them is evidence of meeting a contract rather than proof of correctness.
Why it matters: It keeps evaluation focused on the promises the code must keep, rather than on whether its implementation happens to pass a check. - Limits of enumerated tests: Explain why even many well-written unit tests leave untested inputs, and distinguish exercising code from comprehensively specifying its required behavior.
Why it matters: It explains why a green test suite can still leave important behavior unchecked.
Key terms: tests as contracts, specifications, intent, implementation, unit tests, coverage, specific cases
8.4 Tests that break AI-generated code
The section explains why example-based unit tests share a developer’s blind spots and introduces adversarial verification as a way to find unexpected failures in AI-generated code.
- Example-Based Testing Blind Spots: Explain why passing unit tests does not show that AI-generated code handles inputs the developer never thought to test.
Why it matters: A passing test suite can leave the most surprising failures untouched. - Adversarial Verification Approach: Describe how adversarial verification seeks failures beyond the examples a developer chooses.
Why it matters: It gives verification a chance to expose failures neither the developer nor the AI anticipated.
Key terms: unit tests, example-based testing, blind spots, adversarial verification, properties, randomness, systematic exploration
8.4.1 Property-based testing: from examples to universals
The section explains how property-based testing checks universal claims about a parser, why the choice of property matters, and what random input generation can miss.
- Properties Instead of Examples: Explain how property-based testing differs from example-based testing: the author states a rule that should hold across inputs, and a framework generates inputs to challenge it.
Why it matters: A universal claim can expose failures that hand-picked examples never exercise. - Parser Properties Worth Checking: Describe the parser properties used as examples: identical inputs give identical outputs, plain text without markup passes through unchanged, and arbitrary inputs should produce a string rather than crash. Distinguish these from the proposed well-formed-HTML property.
Why it matters: These claims give the generated inputs something specific to challenge. - Uninformative Passing Properties: Explain why the passing plain-text property offers limited confidence: excluding every markup character restricts the generated inputs to ones the parser ignores.
Why it matters: A large number of passing cases is not reassuring if those cases avoid the behavior under scrutiny. - Random Generation Blind Spots: Explain why the no-crash property passed even against the vulnerable regex: the failure needs a long, monotonous input, while random generation favors shorter, varied strings. State the section's distinction between that approach and fuzzing that mutates toward unexplored paths.
Why it matters: The inputs a generator tends to produce determine which failures a passing test can meaningfully rule out.
Key terms: property-based testing, properties, Hypothesis, deterministic, plain text, well-formed HTML, should not crash, markup characters, counterexample, random generators
8.4.2 Invariants, metamorphic relations, and conservation laws
The section explains three kinds of properties for testing a markup parser and shows how failed property tests can reveal either code defects or missing specifications.
- Input Independent Invariants: Explain that an invariant should hold for every input, and describe balanced output tags as a useful invariant for the parser.
Why it matters: An invariant gives you a check that applies across many generated inputs. - Conditional Metamorphic Relations: Describe how a metamorphic relation connects outputs when inputs are changed, and explain why the parser's concatenation relation fails unless sections are independent.
Why it matters: A failed relation can expose an unstated design choice, not just a coding error. - Conserved Link Counts: Explain conservation laws as properties that preserve a quantity, using the parser's expected one-to-one correspondence between markdown links and HTML links.
Why it matters: Counting what must be preserved can catch whole categories of loss or duplication bugs without claiming complete correctness. - Property Failures Reveal Specifications: Explain why a failing property test can reveal an unresolved definition rather than a faulty parser, using the disagreement over whether []() is a link.
Why it matters: Before fixing a failure, you may need to decide what behavior the software is supposed to have.
Key terms: invariants, unbalanced tags, metamorphic relations, concatenation, multi-line bold, conservation laws, markdown link, HTML link, property-based testing, specification
8.4.3 Fuzzing: when random isn't random enough
The section explains how feedback-guided and LLM-guided fuzzing find hard-to-reach bugs, and when their cost is justified.
- Coverage-guided fuzzing loop: Explain how coverage-guided fuzzing uses feedback from instrumented code to explore inputs likely to reach new branches, rather than relying on random inputs alone.
Why it matters: Feedback lets fuzzing spend more effort on inputs that reveal previously unexplored behavior. - LLM-guided failure inputs: Describe how LLM-guided fuzzing proposes plausible but tricky inputs and why its objective differs from that of an LLM generating code.
Why it matters: A failure-seeking prompt can surface tricky cases that random mutation may take much longer to find. - Fuzzing cost and fit: Explain the setup and runtime costs of fuzzing and distinguish where the section says that investment is most valuable.
Why it matters: The risk posed by untrusted input can justify fuzzing's extra setup and compute time.
Key terms: coverage-guided fuzzing, instrumenting, branches, mutates inputs, LLM-guided fuzzing, plausible-but-tricky inputs, failure-inducing inputs, instrumentation, harness, untrusted input
8.4.4 Mutation testing: testing your tests
The section explains how mutation testing checks whether tests detect meaningful bugs, why conventional mutations can miss AI-generated faults, and what the technique costs.
- Mutation survival and test quality: Explain how mutation testing changes code, reruns the tests, and uses surviving mutations to reveal behaviors the tests do not verify. Distinguish mutation score from code coverage using the parser example.
Why it matters: A passing, high-coverage suite may still let real behavioral changes slip through. - Syntactic and semantic mutations: Explain why conventional small syntactic mutations may fail to represent faults in AI-generated code, and describe the kind of behavior-level mutation an LLM can generate instead.
Why it matters: The bugs used to challenge tests should resemble the mistakes the code generator might actually make. - AI test gaps and mutation cost: Describe why mutation testing is useful when AI generates implementation and tests together, and state its computational trade-off.
Why it matters: Mutation results can direct test improvements, but obtaining them takes substantial test-running time.
Key terms: mutation testing, mutation, survived, killed, mutation score, coverage, competent programmer hypothesis, syntactic mutants, semantic mutation, LLM-powered mutation testing
8.4.5 Differential testing: multiple versions, one truth
This section explains how comparing independent implementations on the same inputs can expose bugs without a known correct answer for every test case.
- Disagreement as a bug signal: Explain how differential testing compares independent implementations of the same specification on generated inputs, and why a disagreement calls for investigation rather than automatically identifying the correct output.
Why it matters: Disagreement gives you a useful test signal even when you cannot write down the expected result for every input. - AI variants and comparison limits: Describe how different models or prompts can supply implementations for differential testing, and distinguish a disagreement that needs human judgment from agreement that may still conceal a shared error.
Why it matters: Several AI-generated answers can expose uncertainty, but their agreement should not be mistaken for proof.
Key terms: differential testing, independent implementations, random inputs, disagreement, different models, different prompts, consensus, shared blind spots, human judgment
8.4.6 New mindset: adversarial verification
The section explains why AI-generated code calls for adversarial verification and where unit-level checks leave a gap.
- Adversarial verification mindset: Explain how adversarial verification differs from checking expected cases, and why the section says it is especially necessary for AI-generated code.
Why it matters: It shifts your question from whether the code passes familiar examples to where it might fail. - Limits of isolated verification: Distinguish what the techniques discussed here check from the system behavior they do not establish.
Why it matters: A component can pass rigorous checks in isolation while the overall system still behaves incorrectly.
Key terms: adversarial mindset, AI-generated code, implicit verification, individual functions or components, system-level contracts, end-to-end behavior
8.5 System-level and contract testing
The section explains why verifying a component in isolation is not enough: its connections and assumptions can fail even when the component itself is correct.
- Failures beyond the component: Explain why a correctly tested parser can still cause problems in a real system, and what system-level verification examines instead.
Why it matters: Passing component tests does not establish that the component works correctly in the system that uses it.
Key terms: system-level verification, contract, system assumptions
8.5.1 Design by contract: making assumptions explicit
The section explains how explicit, executable contracts make the assumptions and guarantees of AI-generated functions visible and catch boundary failures early.
- Explicit function contracts: Explain how Design by Contract turns a function’s implicit assumptions and promises into preconditions, postconditions, and invariants, especially when AI-generated code has no human author holding those assumptions in mind.
Why it matters: Explicit contracts give you a way to state what AI-generated code requires and promises instead of leaving it to guesswork. - Executable boundary checks: Describe how runtime contract checks expose integration errors at a function boundary instead of letting failures propagate into less obvious behavior.
Why it matters: An early failure is easier to diagnose than mysterious behavior deeper in the system. - Verified assumptions and tradeoffs: Explain how writing contracts documents AI-generated code’s requirements and can reveal unanswered questions, while acknowledging the cost of runtime checks and articulating assumptions.
Why it matters: Writing a contract can uncover decisions about edge cases before those ambiguities become production bugs.
Key terms: Design by Contract, preconditions, postconditions, invariants, require decorators, ensure decorators, unstated assumptions, runtime overhead
8.5.2 Contract testing: AI and API stability
The section explains how consumer-owned contract tests catch AI-generated API changes that would break another service’s expectations.
- Consumer and provider contracts: Explain how contract testing checks an API relationship between services: the consumer declares what it will send and expects back, and the provider verifies that it delivers that response.
Why it matters: This checks whether a service still works for its callers, not just whether its own code passes tests. - AI-generated API breakage: Describe why an AI-generated parser change can break another service despite appearing to improve the parser, and how a contract test exposes that break.
Why it matters: A locally successful AI edit can still be a breaking change for a service that depends on the old API. - Independent consumer-owned verification: Explain why Pact-style contracts live with the consumer and are verified independently of provider changes, and state the participation and overhead this requires.
Why it matters: Separation preserves an independent check when an agent might otherwise edit or delete the tests alongside its code.
Key terms: Contract Testing, consumer, provider, Design by Contract, API contract, breaking changes, contract test, Pact-style workflow, consumer's repository, contract broker
8.5.3 Integration and end-to-end tests
Integration and end-to-end tests check that a complete content workflow works for users, even when AI changes its internal implementation.
- Integration and E2E test scope: Distinguish what unit, contract, integration, and end-to-end tests verify. Trace how an integration test checks the parser with surrounding systems, while an E2E test checks the entire user journey.
Why it matters: Code can pass isolated checks yet fail when its parts run together. - Behavior-focused test resilience: Explain why integration and E2E tests are valuable when AI refactors or replaces an implementation: they check the user's observable outcome rather than how the parser works internally.
Why it matters: A surprising AI refactor should be judged by whether the user-facing behavior still works, not merely by whether internals changed.
Key terms: unit tests, contract tests, integration tests, E2E tests, outcomes, implementations, observable behavior, refactoring
8.5.4 Self-healing tests: real value vs. marketing
This section distinguishes useful self-healing of fragile tests from dangerous automatic adaptation to changed application behavior.
- Cosmetic test self-healing: Describe the kinds of test breakage self-healing can usefully repair, such as changed locators, layouts, or minor text, without treating those repairs as proof that the application is correct.
Why it matters: It helps you use automation to remove test maintenance without weakening what the test is meant to check. - Functional failures as review signals: Explain why self-healing a functional test can conceal an AI-generated regression, and why a behavior-changing failure should prompt human review instead of automatic adaptation.
Why it matters: A failing functional test preserves evidence that a change may be wrong, particularly when AI generated the code.
Key terms: self-healing tests, test fragility, locators, intentional change, regressions, business logic verification
8.5.5 Testing what the AI didn't know it changed
This section shows how to detect unintended behavior changes in AI-generated code through characterization testing and targeted impact analysis.
- Unseen behavioral regressions: Explain how an AI change can pass tests for its requested feature yet alter existing behavior the AI did not realize mattered.
Why it matters: Passing the obvious tests does not establish that an AI-generated change preserved important existing behavior. - Characterization tests for preservation: Describe how to capture behavior before a change and compare it afterward, allowing only intended differences.
Why it matters: A record of existing behavior can reveal collateral changes that feature-focused tests miss. - Call graph impact analysis: Trace how impact analysis identifies callers of changed functions, determines the behaviors they rely on, and targets tests to those dependencies.
Why it matters: Targeting dependent behaviors can find integration regressions without testing everything.
Key terms: unknown unknowns, collateral damage, Chesterton's fence, characterization testing, corpus, explicitly allowed patterns, impact analysis, diff, call graph, blast radius
8.6 Formalism and proofs (the high-cost zone)
Introduces formal verification as a costly way to establish that code satisfies a specification, while stressing that even a proof cannot establish that the specification captures the intended behavior.
- Proofs versus sampled tests: Distinguish testing's checks of selected inputs from formal methods' aim of proving correctness for all cases covered by a specification.
Why it matters: This distinction explains what a proof can offer that even extensive testing cannot. - Specification and intent gap: Explain why proving code correct against a specification does not prove that the specification expresses what people intended, including when AI drafts it.
Why it matters: Proven code can still faithfully implement the wrong requirement. - Cost and criticality tradeoff: Justify why formal methods are usually excessive for ordinary projects but may merit attention for code involving money, security, or human safety.
Why it matters: The value of stronger assurance has to be weighed against the effort it takes to obtain it.
Key terms: testing, sampling, formal methods, formal proof, specification, spec, formal verification
8.6.1 Much more strong(er) type systems
This subsection shows how stronger type systems can turn correctness constraints for AI-generated code into compile-time guarantees.
- Refinement Types as Compile-Time Guarantees: Explain how refinement types add constraints to ordinary types and let the compiler reject code that could violate them, including AI-generated code that might otherwise require a test to catch.
Why it matters: A compiler-enforced constraint can catch a possible AI-generated bug before the code runs. - Value-Dependent Type Relationships: Describe how dependent types express a relationship between input and output values as a compiler-verified type constraint.
Why it matters: They can make a promised relationship between a function's input and output compiler-checkable.
Key terms: refinement types, compile-time proofs, strong type systems, dependent types, type-level constraint
8.6.2 SMT solvers: automated theorem proving
The section explains how SAT and SMT solvers can verify code properties by searching for satisfying inputs, and why complex code limits their practicality.
- Constraint-based code verification: Explain how a SAT or SMT solver uses logical constraints to check a code property, and what “sat” and “unsat” mean for the property being checked.
Why it matters: A solver can turn a precise concern about code into either a counterexample or a proof that no counterexample exists. - SMT scalability limits: Describe why SMT verification becomes difficult as code grows more complex, while noting that tools such as Z3 and CVC5 make it accessible.
Why it matters: Solver results can be powerful, but the complexity of the code determines whether this kind of verification is practical.
Key terms: SAT, SMT, logical constraints, unsat, SMT solvers, logical formulas, Z3, CVC5, scalability
8.6.3 LLM + symbolic execution: the emerging hybrid
The section explains how LLMs can guide symbolic execution toward promising code paths while formal analysis supplies rigor, though the hybrid remains research-grade.
- Symbolic Execution and Path Explosion: Explain how symbolic execution follows a symbolic input through a program and why examining every path is infeasible.
Why it matters: Path explosion explains why symbolic execution benefits from guidance about where to look. - LLM-Guided Formal Exploration: Describe how an LLM guides symbolic execution toward likely bug-revealing paths, what each technique contributes, and the approach's current maturity.
Why it matters: The hybrid shows how AI's guidance and formal methods' rigor can complement each other without implying the tooling is ready for routine use.
Key terms: symbolic execution, symbolic values, constraints, path explosion, LLM-guided symbolic execution, LLM-Sym, formal methods, research-grade tooling
8.6.4 Formal verification with AI assistance
This section explains how formal verification works, why its specification burden limits adoption, and how AI can help without replacing human judgment or mathematical verification.
- Formal proof and verification tools: Distinguish formal verification from testing and type checking, and describe what TLA+ and Dafny verify in this section's examples.
Why it matters: A proof is meaningful only in relation to the specification being checked. - Specification and annotation burden: Explain why formal verification has remained niche despite its power, including the risk of outdated specifications.
Why it matters: The cost and upkeep of specifications explain why powerful verification tools are not used everywhere. - AI-assisted verification responsibilities: Describe how AI assists formal verification, what the verifier and human must still do, and where the section recommends using this approach.
Why it matters: AI can lower the entry barrier, but neither a plausible AI-written specification nor a proof of the wrong specification establishes the intended behavior.
Key terms: formal verification, formal specification, TLA+, Dafny, preconditions, postconditions, annotation burden, loop invariants, intermediate assertions, specifications
8.6.5 When to consider such a level of evaluation
This section explains when formal methods are worth their cost and why they can complement AI-generated code.
- Formal verification decision criteria: Explain why formal methods are overkill for most code and describe the circumstances that can justify them: catastrophic bugs, naturally formal domains, persistent subtle corner cases, or regulatory demands.
Why it matters: The right level of verification depends on the risk, the code, and the cost of checking it. - Complementary AI and formal methods: Explain why the section presents AI and formal methods as complementary, and why their practicality is worth revisiting.
Why it matters: Verification techniques that once seemed too costly may become practical ways to build confidence in AI-generated code.
Key terms: formal methods, property-based testing, mutation testing, contract testing, formal verification, AI plus formal methods, rigorous verification, LLM-generated code
8.7 Runtime verification and production
The section explains why pre-deployment tests cannot anticipate every real-world behavior and why deliberate production-time verification is necessary.
- Limits of pre-deployment testing: Explain why even extensive testing cannot rule out production surprises: tests rely on simplified models of users, data, load, and environments.
Why it matters: It explains why confidence cannot end when the pre-deployment tests pass. - Deliberate runtime verification: Describe runtime verification as using production deliberately to detect, contain, and learn from failures that could not be fully verified beforehand.
Why it matters: It turns inevitable production surprises into something a team can identify and respond to.
Key terms: testing operates on models, production, runtime verification, detect, contain, and learn
8.7.1 Regression testing as CI for every AI change
The section explains why every AI-generated change should pass regression tests in CI and how to choose the scope of those tests.
- Unpredictable AI change impact: Explain why even a small request to an AI coding tool warrants regression testing in CI: the resulting code change may be much broader than requested, making its blast radius hard to predict.
Why it matters: The size of the request is not a reliable guide to how much testing the change needs. - Regression test scope tradeoff: Describe the two CI testing approaches presented: tag tests by component and dependency to run potentially affected tests, or run the entire suite for every AI change. Explain the extra CI time versus the chance of catching unexpected regressions.
Why it matters: Choosing CI test scope means balancing faster feedback against the risk of missing effects you did not anticipate.
Key terms: blast radius, regression testing, CI/CD pipelines, component and dependency, entire test suite, targeted test selection, CI times
8.7.2 Observability as evaluation: logging, tracing, and metrics
The section shows how production logs, traces, and metrics evaluate AI-generated code against real usage and expose cases that pre-production tests missed.
- Production logs as test inputs: Explain how logging unusual, slow, or error-triggering parser inputs reveals cases missed by testing and supplies inputs for further investigation and tests.
Why it matters: Real user inputs can reveal cases that planned tests missed. - Request tracing for diagnosis: Describe how tracing follows a request through services, parser, and renderer to show where an incorrect result arose, even when AI-generated code is hard to reason about.
Why it matters: A trace can pinpoint where a real request went wrong without requiring complete understanding of the generated code. - Metrics as production signals: Explain how aggregate production metrics reveal system health and changes after deploying AI-generated code, making real usage an ongoing source of evaluation.
Why it matters: Aggregate signals can reveal regressions that only appear under real usage.
Key terms: logging, property test corpus, tracing, flow of requests, metrics, p50, p95, p99, error rate, observability
8.7.3 Canary deployments and progressive rollout
The section explains why generated code needs more cautious production rollouts and how A/B testing answers a different question from correctness checks.
- Cautious rollout of generated code: Explain why staged rollout carries more weight for generated code than for human-written code, and describe how the rollout should change.
Why it matters: A staged rollout helps catch behavior that was never considered during generation. - Production comparison through A/B testing: Distinguish the question A/B testing answers from the correctness question answered by other evaluation methods in the chapter.
Why it matters: Code can be correct without improving on what it replaces.
Key terms: canary deployments, progressive rollout, feature flags, staging environment, ramp, A/B testing, generated parser, hand-written one, production
8.7.4 Chaos engineering: testing resilience
The section explains how deliberately introduced failures reveal whether AI-generated code fails safely under adverse conditions.
- Safe failure under stress: Explain how chaos engineering introduces failures to check whether AI-generated code degrades gracefully rather than corrupting state, hanging, or crashing.
Why it matters: Code that works in normal conditions can still damage or stall a system when resources or dependencies fail.
Key terms: chaos engineering, fail safely, degrade gracefully, resource exhaustion, happy-path
8.7.5 Non-functional testing: performance, security, accessibility, compatibility
The section explains why code that passes functional tests still needs performance, security, accessibility, and compatibility testing.
- Non-functional quality checks: Explain how non-functional tests assess qualities that functional tests can miss, and describe what performance, security, accessibility, and compatibility tests check in AI-generated code.
Why it matters: Passing functional tests does not establish that AI-generated code is viable or safe to use.
Key terms: functional tests, non-functional tests, performance testing, security testing, accessibility testing, compatibility testing
8.8 Choosing evaluation methods
This section frames evaluation as choosing among verification techniques rather than applying every technique to every project.
- Selective verification choices: Explain why evaluation requires choosing which verification techniques to apply where, rather than using every available technique on every project.
Why it matters: Selective evaluation helps you build confidence without treating every possible check as mandatory.
Key terms: verification, always-on gates, runtime observability
8.8.1 What determines "sufficient confidence"?
The section explains that sufficient confidence in code depends on the consequences of failure and the effort justified to reduce that risk.
- Risk-calibrated confidence in code: Explain how the cost of incorrect code determines the confidence needed and the evaluation effort worth investing. Illustrate the difference using the parser in a personal blog, a news organization, and a browser-facing application.
Why it matters: It helps you spend evaluation effort where a failure would do the most harm.
Key terms: sufficient confidence, property-based tests, integration tests, security testing, fuzzing, XSS attacks
8.8.2 System criticality vs. evaluation cost
The section explains how to spend verification effort according to the cost of bugs, while keeping inexpensive checks as a baseline for all code.
- Risk-based verification investment: Explain why verification effort should rise with system criticality: minimal effort for low-cost failures, standard verification for most production features, and intensive verification when failures could be severe.
Why it matters: The right amount of verification depends on what a failure would cost. - Verification methods by risk: Apply the section's method choices: use cheap always-on gates for all code, adversarial checks for untrusted input or complex logic, system-level checks for integrations, and formal methods or runtime verification for critical code.
Why it matters: Choosing checks by the failures they can catch avoids both wasted effort and dangerous gaps.
Key terms: system criticality, verification investment, risk management, always-on gates, adversarial verification, system-level verification, formal methods, runtime verification
8.8.3 When to test more vs. when to regenerate
This section explains when regenerating AI-generated code is more sensible than debugging it, and when existing test evidence should guide the decision instead.
- Regeneration versus debugging cost: Explain why regenerating AI-generated code can be a rational alternative to debugging it.
Why it matters: The cheaper path to a verifiable result is not always the path that preserves the first draft. - Debug regenerate or trust tests: Apply the section's decision framework to choose whether to debug, regenerate, step back, or rely on test evidence.
Why it matters: These distinctions keep a team from spending hours on opaque code or discarding code without evidence of a problem.
Key terms: debugging, regeneration, mental model, debug, regenerate, prompt, tests are evidence
8.8.4 Minimum viable testing for AI-first teams
The section lays out a minimum viable verification stack for teams that generate significant code with AI, organized by when checks should run.
- Minimum Viable Verification Cadence: Describe the four verification cadences for an AI-first team and the checks assigned to each: every change, before merge, regularly, and before major releases.
Why it matters: Different checks catch different problems, so the team needs a repeatable schedule rather than relying on a single passing test run.
Key terms: minimum viable verification stack, Always on, Before merge, Regularly, Before major releases
8.8.5 Evaluation as competence, not checklist
The section argues that verifying AI-generated code requires human judgment beyond completing a testing checklist.
- Checklists as a Verification Floor: Explain why a verification checklist is useful but insufficient for deciding whether AI-generated code is trustworthy.
Why it matters: Completing the prescribed checks is a starting point, not proof that the code is sound. - Human Evaluation Competence: Describe the judgment needed to evaluate AI-generated code, including understanding its behavior, questioning the tests, and being candid about remaining uncertainty.
Why it matters: When AI writes the code, your responsibility is to judge whether the evidence warrants trusting it.
Key terms: checklist, verification, judgment, evaluation competence, mental model, coverage numbers, false confidence
8.8.6 Revenge of the QA
This section argues that as AI automates code generation, software engineering increasingly centers on QA-style evaluation and the ability to establish trust in code.
- QA Skills Become Central: Explain why automated code generation raises the value of systematic testing, adversarial thinking, edge-case intuition, and failure-mode analysis.
Why it matters: It reframes testing expertise as central engineering work rather than a secondary check. - Trust Engineering Mindset: Describe the shift from producing code to deciding whether code deserves trust, including the move from expected behavior to probing failure paths and judging their risks.
Why it matters: The engineer’s responsibility is to establish confidence in system behavior regardless of who—or what—wrote the code.
Key terms: QA, evaluation, adversarial thinking, failure mode analysis, trust engineer, happy paths, failure paths, build confidence
8.9 Summary
Where this is used later
- The uncomfortable truth about this book (chapter 10) · The distinction between generating code and gaining confidence in it.
- The three lies (chapter 10) · The distinction between tests that describe intent and tests that reflect implementation.
- Fix the lies before you fix the speed (chapter 10) · Mutation testing as a way to challenge a test suite.
Concepts explained in this chapter
- verificationThe systematic evaluation of an implementation or output against specified requirements and expected behavior.
- software testingAn executable procedure that evaluates software behavior against expected results or conditions.
- property-based testingA testing technique that checks general properties or invariants across many generated or varied inputs rather than only fixed examples.
- fuzzingFuzzing is an automated testing technique that supplies large numbers of generated, malformed, or unexpected inputs to discover failures and vulnerabilities.
- mutation testMutation testing evaluates a test suite by introducing small controlled changes into code and checking whether the tests detect them.
- invariantA software invariant is a property or condition that must remain true throughout specified program states or operations.
- redosA regular-expression denial-of-service vulnerability causes excessive regex matching time, often through catastrophic backtracking on attacker-controlled input.
- static analysisThe automated examination of source code without executing it to identify defects, suspicious constructs, policy violations, or program properties.
- snapshot comparisonA golden file is an approved reference-output file against which subsequent program output is compared.
- unit testA unit test evaluates an isolated software unit, such as a function or class, against expected behavior.
- AI-generated pull requestSource code produced in whole or in part by an artificial-intelligence system.
- judgmentHuman judgment is the reasoning and decision-making people apply when evaluating situations, evidence, risks, and software changes.
- adversarial thinkingAdversarial verification is verification that deliberately probes software with hostile, unexpected, or broadly varied inputs to expose weaknesses.
- specificationA statement of required system behavior, constraints, or properties that defines what an implementation should satisfy.
- test-driven promptingTest-Driven Prompting is an AI-assisted development technique that supplies behavioral tests as specifications for generating code.
- mental modelAn internal representation of how a system works that supports reasoning about its behavior and consequences.
- formal verificationThe use of mathematical specifications and proofs to establish that software satisfies required properties for all permitted cases.
- code coverageCode coverage is a metric reporting the proportion of source-code lines, branches, or paths executed during testing.
- parserA program component that analyzes an input sequence according to a formal grammar and produces a structured representation.
- formal methodFormal methods are mathematically rigorous techniques for specifying, modeling, analyzing, and verifying properties of software or hardware systems.
- contract testingA testing technique in which a service consumer's expected requests and responses are checked against a service provider's actual behavior.
- contractAn API is a defined interface through which software systems communicate and invoke one another's capabilities.
- code reviewThe manual examination of proposed source-code changes to identify defects, risks, and conformance or maintainability problems before integration.
- deployment pipelineThe practice of continuously integrating code changes and continuously delivering or deploying validated software through automated pipelines.
- lintingA local linter is a tool run in a developer's environment that statically analyzes source code for style issues, likely errors, and convention violations.
- static application security testingStatic Application Security Testing is the analysis of source code or compiled code without executing the application to identify security vulnerabilities.
- benchmarkA benchmark is a standardized evaluation procedure that measures a system against defined inputs and expected results.
- AI-assisted code reviewAn AI review agent is an AI system that analyzes proposed software changes and reports violations, risks, or issues requiring human judgment.
- checklistA checklist is a structured list of specific items or conditions to inspect when evaluating an artifact or process.
- cross-model validationCross-model validation is the practice of having a model different from the generating model evaluate generated output.
- manual testingManual testing is the evaluation of software behavior by a human executing and inspecting test procedures rather than automated test code.
- characterization testingA testing technique that records existing software behavior and checks that later changes preserve it except for explicitly intended differences.
- regular expressionA regular expression is a pattern notation for matching and extracting text according to specified character and sequence rules.
- observabilityAutomated observability is the machine-driven collection and analysis of signals that make a system's behavior, failures, and quality degradation detectable.
- integration testAn integration test evaluates interactions between multiple software components or with external infrastructure rather than isolating a single unit.
- accuracyCorrectness is the degree to which software behavior satisfies its intended specification and real-world requirements.
- trust deficitA trust deficit is the gap between the speed at which AI produces code and the ability to adequately verify that code.
- production-ready softwareThe state in which software is adequately reliable, operable, secure, and resilient for real-world deployment.
- dependencyA software dependency is a component, package, library, or other resource that another component requires to function correctly.
- slopsquattingAn attack in which an attacker registers package names that language models are likely to hallucinate so developers may install malicious software.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
