An evaluation suite is a collection of automated tests that checks software behavior against specified requirements. It can serve as an executable contract for AI-generated code: the implementation must satisfy the tests rather than merely look plausible. Passing the suite establishes agreement with the checks it contains, so the quality of those checks matters. section 1.4 section 1.8
An ISBN validation contract
A validator may look syntactically sound yet reject valid hyphenated ISBNs. A human-authored pytest suite checks both formatted and unformatted valid inputs, plus invalid length, non-numeric characters, and an incorrect checksum. Three generated implementations passed the same suite despite stylistic differences; the common contract made their behavior comparable. section 1.4
import pytest
# Assume is_valid_isbn13 is the function to be generated
def test_valid_isbn_with_hyphens():
assert is_valid_isbn13("978-0-306-40615-7") is True
def test_valid_isbn_without_hyphens():
assert is_valid_isbn13("9780306406157") is True
def test_invalid_checksum():
assert is_valid_isbn13("978-0-306-40615-0") is False
def test_invalid_length():
assert is_valid_isbn13("978-0-306-40615") is False
def test_contains_non_numeric_characters():
assert is_valid_isbn13("978-0-306-40615-X") is FalseThe pytest listing makes the ISBN requirements executable as five assertions. section 1.4
Run regression tests for every AI-generated change, even when the request seems small: a generated edit may rewrite more than expected. Tag tests by component and dependency so that a parser change also triggers potentially affected service, rendering, and API tests. Running the entire suite is a more conservative alternative; it costs more CI time but can catch regressions that targeted selection misses. section 8.7.1





















