Characterization testing records existing behavior before a change, then rejects unexpected differences afterward. It protects against collateral damage when an AI-generated modification changes behavior no one intended to touch. section 8.5.5
Apply it as a behavioral snapshot and verification loop:
- Assemble a corpus from production examples, known edge cases, and regressions from past bugs. section 8.5.5
- Run the current implementation and save each input-output result as a snapshot. section 8.5.5
- After the change, rerun the corpus and fail any difference that does not match an explicitly allowed pattern. section 8.5.5
- Treat the result as stability evidence, not proof that the implementation is correct. section 8.5.5
Markup parser snapshot
Capture parser outputs for markup samples in a characterization snapshot before an AI change. Re-run the same samples afterward; an output change fails unless its input matches an approved pattern such as an intentional strikethrough modification.
Listing 8.20 Characterization test: capture and verify behavior
def generate_characterization_tests():
"""Run before AI makes changes"""
corpus = load_corpus('markup_samples.txt')
results = {input: parse_markup(input) for input in corpus}
save_snapshot('parser_characterization.json', results)
def verify_characterization(allowed_changes=['~~.*~~']):
"""Run after AI makes changes"""
corpus = load_corpus('markup_samples.txt')
old_results = load_snapshot('parser_characterization.json')
for input, old_output in old_results.items():
new_output = parse_markup(input)
if old_output != new_output:
if not matches_allowed_pattern(input, allowed_changes):
raise AssertionError(
f"Unexpected behavior change:\n"
f"Input: {input}\n"
f"Old: {old_output}\n"
f"New: {new_output}"
)
Inspect the capture-and-verify listing for the snapshot workflow and explicit exception handling. section 8.5.5

