liveBook · Manning

Property-based testing

As explained in Vibe Engineering

A testing technique that checks general properties or invariants across many generated or varied inputs rather than only fixed examples.

A test suite can pass its fixed examples while missing inputs that break the implementation. Property-based testing instead states properties that should hold across a range of inputs and lets a framework generate cases that try to violate them. It is an adversarial check, not proof that every input is correct. section 8.4.1 section 8.4.6

Choose properties that make claims about behavior that matters. A parser should be deterministic, return a string without crashing, and produce well-formed output. A round-trip property can be useful when an inverse function exists: converting parsed output back to markup should recover valid input. section 8.4.1

Listing 8.8 Property test: parser is deterministic
from hypothesis import given
from hypothesis.strategies import text
 
@given(text())
def test_deterministic(s):
    assert parse_markup(s) == parse_markup(s)

This Hypothesis test generates strings and checks whether parsing the same input twice gives the same result. Hidden state, randomness, or race conditions can make that assumption fail. section 8.4.1

Passing tests can be misleading if the input strategy excludes the behavior under scrutiny. A plain-text pass-through test filters out every markup character the parser reacts to; it checks ignored inputs, not markup handling. section 8.4.1

Several kinds of properties help expose different failures:

  • Invariants hold for each input, such as a parser returning without crashing or producing balanced tags. Simple tag-count checks are useful but do not fully establish well-formed HTML. section 8.4.2
  • Metamorphic relations compare results after a controlled input change. Parsing two lines separately might match parsing them together, but only if the syntax makes lines independent. section 8.4.2
  • Conservation laws check quantities that should be preserved, such as one output link for each recognized input link. The counting rule must match the parser's actual definition of a link. section 8.4.2
Listing 8.12 Metamorphic relation: concatenation independence
@given(text(), text())
def test_concatenation(s1, s2):
    # This assumes lines are independent
    combined = s1 + "\n" + s2
    separate = parse_markup(s1) + "\n" + parse_markup(s2)
    assert parse_markup(combined) == separate

This concatenation test makes line independence explicit. A failure can reveal cross-line markup behavior; decide whether the implementation is wrong or the assumed property is wrong. section 8.4.2

Use property-based tests alongside unit and integration tests, not as a replacement. Prioritize complex logic and untrusted inputs, and scale verification effort to the cost of failure. For teams generating substantial code with AI, run property tests on core logic regularly and keep integration checks before merge. section 8.8.2 section 8.8.4

How Vibe Engineering covers it

How deep it goes
Works through generated parser properties and their assumptions, then defers system integration behavior to contracts and integration tests.
Examples
Tests a markup parser for determinism, plain-text preservation, tag balance, line independence, and link-count preservation.
What you can do after
Apply it via Hypothesis to generate parser inputs and investigate whether failures expose implementation bugs or incorrect property assumptions.
How it connects
Connects invariants, metamorphic testing, and conservation laws with fuzzing, mutation testing, and contract testing.

Also explained in

Compare how 11 books explain property-based testing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.