7 Trustworthy tests

 

This chapter covers

  • How to know you trust a test
  • Detecting untrustworthy failing tests
  • Detecting untrustworthy passing tests
  • Dealing with flaky tests

No matter how you organize your tests, or how many you have, they’re worth very little if you can’t trust them, maintain them, or read them. The tests that you write should have three properties that together make them good:

  • Trustworthiness—Developers will want to run trustworthy tests, and they’ll accept the test results with confidence. Trustworthy tests don’t have bugs, and they test the right things.
  • Maintainability—Unmaintainable tests are nightmares because they can ruin project schedules, or they may be sidelined when the project is put on a more aggressive schedule. Developers will simply stop maintaining and fixing tests that take too long to change or that need to change often on very minor production code changes.
  • Readability—This refers not only to being able to read a test but also figuring out the problem if the test seems to be wrong. Without readability, the other two pillars fall pretty quickly. Maintaining tests becomes harder, and you can’t trust them anymore because you don’t understand them.

This chapter and the next two present a series of practices related to each of these pillars that you can use when doing test reviews. Together, the three pillars ensure your time is well used. Drop one of them, and you run the risk of wasting everyone’s time.

7.1 How to know you trust a test

7.2 Why tests fail

7.2.1 A real bug has been uncovered in the production code

7.2.2 A buggy test gives a false failure

7.2.3 The test is out of date due to a change in functionality

7.2.4 The test conflicts with another test

7.2.5 The test is flaky

7.3 Avoiding logic in unit tests

7.3.1 Logic in asserts: Creating dynamic expected values

7.3.2 Other forms of logic

7.3.3 Even more logic