liveBook · Manning

Test suite

As explained in Vibe Engineering

A collection of automated tests that evaluates software behavior against specified requirements.

An evaluation suite is a collection of automated tests that checks software behavior against specified requirements. It can serve as an executable contract for AI-generated code: the implementation must satisfy the tests rather than merely look plausible. Passing the suite establishes agreement with the checks it contains, so the quality of those checks matters. section 1.4 section 1.8

An ISBN validation contract

A validator may look syntactically sound yet reject valid hyphenated ISBNs. A human-authored pytest suite checks both formatted and unformatted valid inputs, plus invalid length, non-numeric characters, and an incorrect checksum. Three generated implementations passed the same suite despite stylistic differences; the common contract made their behavior comparable. section 1.4

import pytest
# Assume is_valid_isbn13 is the function to be generated
def test_valid_isbn_with_hyphens():
    assert is_valid_isbn13("978-0-306-40615-7") is True
def test_valid_isbn_without_hyphens():
    assert is_valid_isbn13("9780306406157") is True
def test_invalid_checksum():
    assert is_valid_isbn13("978-0-306-40615-0") is False
def test_invalid_length():
    assert is_valid_isbn13("978-0-306-40615") is False
def test_contains_non_numeric_characters():
    assert is_valid_isbn13("978-0-306-40615-X") is False

The pytest listing makes the ISBN requirements executable as five assertions. section 1.4

Run regression tests for every AI-generated change, even when the request seems small: a generated edit may rewrite more than expected. Tag tests by component and dependency so that a parser change also triggers potentially affected service, rendering, and API tests. Running the entire suite is a more conservative alternative; it costs more CI time but can catch regressions that targeted selection misses. section 8.7.1

How Vibe Engineering covers it

How deep it goes
Details requirement-focused cases, dependency-based test selection, and the tradeoff between targeted and full-suite CI runs.
Examples
Tests ISBN validation with hyphenated and unformatted inputs, invalid length, non-numeric characters, and an incorrect checksum.
What you can do after
Apply it via pytest to specify ISBN validation behavior and check generated implementations against the same assertions.
How it connects
Connects executable specifications with regression testing, continuous integration, and test-driven development.

Also explained in

Compare how 22 books explain test suite →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.