liveBook · Manning

Characterization testing

As explained in Vibe Engineering

A testing technique that records existing software behavior and checks that later changes preserve it except for explicitly intended differences.

Characterization testing records existing behavior before a change, then rejects unexpected differences afterward. It protects against collateral damage when an AI-generated modification changes behavior no one intended to touch. section 8.5.5

Apply it as a behavioral snapshot and verification loop:

  • Assemble a corpus from production examples, known edge cases, and regressions from past bugs. section 8.5.5
  • Run the current implementation and save each input-output result as a snapshot. section 8.5.5
  • After the change, rerun the corpus and fail any difference that does not match an explicitly allowed pattern. section 8.5.5
  • Treat the result as stability evidence, not proof that the implementation is correct. section 8.5.5

Markup parser snapshot

Capture parser outputs for markup samples in a characterization snapshot before an AI change. Re-run the same samples afterward; an output change fails unless its input matches an approved pattern such as an intentional strikethrough modification.

Listing 8.20 Characterization test: capture and verify behavior
def generate_characterization_tests():
    """Run before AI makes changes"""
    corpus = load_corpus('markup_samples.txt')
    results = {input: parse_markup(input) for input in corpus}
    save_snapshot('parser_characterization.json', results)
 
def verify_characterization(allowed_changes=['~~.*~~']):
    """Run after AI makes changes"""
    corpus = load_corpus('markup_samples.txt')
    old_results = load_snapshot('parser_characterization.json')
    
    for input, old_output in old_results.items():
        new_output = parse_markup(input)
        if old_output != new_output:
            if not matches_allowed_pattern(input, allowed_changes):
                raise AssertionError(
                    f"Unexpected behavior change:\n"
                    f"Input: {input}\n"
                    f"Old: {old_output}\n"
                    f"New: {new_output}"
                )
 

Inspect the capture-and-verify listing for the snapshot workflow and explicit exception handling. section 8.5.5

How Vibe Engineering covers it

How deep it goes
Defines corpus capture, snapshot comparison, and allowed exceptions, but stops short of proving implementation correctness.
Examples
Uses production inputs, edge cases, historical regressions, and markup-parser snapshots with explicitly allowed output changes.
What it assumes
Requires an executable behavior to observe, representative inputs, and a way to save and compare results.
What you can do after
Diagnose unexpected behavior changes by comparing pre-change and post-change outputs across a representative corpus.
How it connects
Connects behavioral stability with regression testing, impact analysis, unit testing, and runtime observability.

Also explained in

Compare how 2 books explain characterization testing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.