liveBook · Manning

Mutation test

As explained in Vibe Engineering

Mutation testing evaluates a test suite by introducing small controlled changes into code and checking whether the tests detect them.

A passing test suite can execute code without checking its behavior. Mutation testing probes that weakness by making small, controlled changes to code and running the tests against each changed version. section 8.4.4

A mutation is killed when the tests fail and survives when they still pass. Surviving mutations point to behavior the suite did not distinguish. Common changes include replacing > with >=, changing + to -, reversing a Boolean, or deleting a statement. section 8.4.4

Listing 8.14 Examples of code mutations
@given(text())
# Original
if line.startswith('###'):
    
# Mutant 1: boundary change
if line.startswith('##'):
 
# Mutant 2: negation
if not line.startswith('###'):
 
# Mutant 3: statement deletion
# (line deleted entirely)

These parser mutations probe whether tests distinguish header levels and detect a deleted header check. section 8.4.4

Coverage without verification

A parser suite reaches 95% coverage but a 67% mutation score: it executes most code while letting roughly a third of the mutations survive. If a changed header check survives, add an assertion that distinguishes the expected header behavior, rather than relying on execution alone. section 8.4.4

The distinction matters when AI generates both implementation and tests: the tests may follow the happy path or mirror what the implementation does instead of checking what it should do. Rewrite weak assertions against the product specification, then rerun the mutations. section 8.4.4 section 10.4.2

Trading coverage for stronger assertions

In a checkout module, 23% of mutations initially survived. After tests were rewritten against the product specification, coverage fell from 94% to 87%, while mutation survival fell to 6%. The lower coverage figure accompanied more effective checks against the mutations. section 10.4.2

Use mutation testing as one adversarial check, not a substitute for system-level verification. Focus it on critical paths or complex logic and untrusted input; run it regularly, such as weekly or per sprint, where the risk justifies the cost. section 8.8.2 section 8.8.4 section 8.4.6

How Vibe Engineering covers it

How deep it goes
Explains mutation outcomes and common operators through parser and production examples, while leaving system integration to other verification techniques.
Examples
Uses header-parser mutations and a checkout suite rewritten against its product specification.
What you can do after
Evaluate test quality by interpreting surviving mutations and strengthening assertions against expected behavior.
How it connects
Connects mutation testing to code coverage, property-based testing, fuzzing, and contract testing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.