liveBook · Manning

False positive

As explained in The Art of Unit Testing 3e

A false positive, or false test failure, is a reported test failure even though the underlying production behavior is correct.

A test fails, but the production behavior is correct. That is a false positive, also called a false test failure. A true failure, by contrast, exposes a production bug. The distinction matters because a failing test is a warning to investigate, not proof that production code is wrong. section 8.1 section 7.2

When a failure looks suspicious, check whether the test is failing for a reason unrelated to a production bug:

  • Is the assertion checking the wrong result, or is the test calling the code incorrectly or supplying the wrong input? section 7.2.2
  • Has the intended behavior or production API changed while the test still expects the old usage? section 7.2 section 8.1.2
  • Does the test conflict with another test, or does it fail intermittently? section 7.2

A string assertion that survives harmless changes

Suppose an error message must contain a reason, but its surrounding whitespace or decoration may change. `expect(errors[0]).toContain('fake reason');` checks the important content without requiring an exact string match. That makes a harmless formatting change less likely to trigger a false positive. section 2.5.4

If the test itself is buggy, fix it and run it again. Then temporarily put an obvious bug in the production code that this test should catch. Confirm the test fails, remove the deliberate bug, and confirm it passes. Passing alone is not enough: the test also needs to fail when the behavior it protects is broken. section 7.2.2

Track false failures and their causes over time to gauge test maintainability. Reduce avoidable failures by asserting observable behavior rather than private details, avoiding unnecessary exact matches, and using factory or helper functions to absorb API changes. The goal is not a test that never fails; it is one whose failures deserve attention. section 8.1 section 8.4

How The Art of Unit Testing 3e covers it

How deep it goes
Gives diagnostic checks and a repair procedure, then identifies maintainability practices that prevent recurring false failures.
Examples
Uses a partial string assertion and constructor-based tests affected by a production API change.
What it assumes
Introduces true versus false failures while using familiar test assertions and function calls in its examples.
What you can do after
Diagnose/debug with it by inspecting a failing test, then injecting and removing an obvious production bug to verify its behavior.
How it connects
Connects false positives to flaky tests, test maintainability, overspecification, and factory functions.

Also explained in

Compare how 26 books explain false positive →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of The Art of Unit Testing 3e. The section links point to the parts of the book it draws on.