Chapter 11: Data Oriented Testing
The chapter introduces data-oriented testing and emphasizes that how tests represent data affects their clarity.
After this chapter you will be able to
- Representation in test code. Explain why representation matters in tests: even small changes to how test data is represented can substantially affect how understandable a test is.
- Unit and integration test boundary. Distinguish unit tests from integration tests using the book’s definitions: unit tests run in isolation, with dependencies mocked if needed; integration tests involve dependencies and IO. Explain why the size or role of a test does not determine its category.
- Input and output testing. Explain why pure, deterministic functions are straightforward to test: a test supplies data, calls the function, and asserts on the returned data.
- Dependency setup and test brittleness. Distinguish tests of isolated behavior from tests of code that mixes business logic with dependencies and side-effects.
- Generated data and deep comparison. Describe how generating valid Account states and CSV from them strengthens the parsing test, and what its final assertion checks.
- Tests as domain explanations. Explain why a good test suite does more than verify behavior and guard against regressions: it shows how the code interacts with the world and which invariants should hold.
- Signs of ineffective tests. Describe why tests that are brittle, hard to interpret, or narrowly tied to one code path are weak even if they satisfy a coverage metric.
- Input and assertion relationship. Explain why a test should show the relationship between its inputs and assertions, rather than leaving the reader to investigate an unexplained expected value.
- Locally visible test state. Describe why relevant test state belongs in the test body and how factory functions can keep that state readable.
- Flexible threshold-driven test inputs. Explain how deriving a test’s inputs from a threshold makes it easy to try different values and uncover assumptions tied to hard-coded constants.
- Weakness of count assertions. Explain why checking only that two sets have the same size can miss an account-filtering bug.
- Generated subset test scenarios. Describe how computing the power set of test data lets a test exercise every subset, including the empty collection, instead of selecting variations by hand.
- Data-driven side-effect verification. Explain how a test can use data to specify an expected side-effect and then assert that it happened, without making mocking the focus of the test.
- Domain states as test data. Explain how introducing data types can make tests describe the meaningful states and lifecycles of a system, using the report-mirroring service as an example.
- Hidden shared-data coupling. Trace how a test that reuses a helper for one needed Account value can silently depend on other hard-coded values outside the test, and explain why changes to shared data can eventually break many tests.
- Shared helpers without hard coding. Explain why the section rejects a blanket ban on shared test data and instead proposes keeping helpful shared functions while removing the hard-coded data that creates dangerous coupling.
- Random inputs and test coupling. Explain why test-data helpers should randomize attributes a test does not need instead of supplying hard-coded values.
- Assertions across unspecified inputs. Explain what randomizing an unspecified attribute says about the test’s assertion and why that matters as software changes.
- Randomized Testing Perspective. Explain how randomized inputs change what a test tries to establish compared with a hard-coded example, and state the limit of that claim.
- Stream-Based Random Test Data. Explain how Stream.generate turns a function that produces one randomized value into a source of arbitrary amounts of test data, including collections tailored to a test.
- Random-Sized and Nested Inputs. Explain why tests can randomize the amount of generated data, including zero, and nest generators to create varied collections of collections.
- State Coverage Through Randomization. Explain why tests should be judged by the states their assertions exercise, not just by whether a few chosen examples pass. Describe how randomized thresholds broaden the test’s intent while a fixed region can still leave important states untested.
- Invariant Guided Data Generation. Explain how to steer random generators toward states needed for an invariant without specifying every input. Use the account’s relationship to a threshold to illustrate which spend values matter.
- Generated input scale and test roles. Explain how changing the amount of generated data lets the same test serve as a unit test or a load test, and how larger runs can reveal a service’s limits.
- Randomized Boundary Round Trips. Explain how repeatedly round-tripping randomized data through boundaries such as a REST API or database can reveal serialization problems, including problems in dependencies, before users encounter them.
- Randomized checks of algebraic laws. Explain how to test an algebraic law when inputs such as Strings or BigDecimals make exhaustive testing impractical, and distinguish growing confidence from proof.
- Randomization and determinism tradeoff. Explain why randomized test data is not suitable for every test: it trades away some determinism, while hard-coded data and mocks are often used precisely to make tests behave predictably.
- Exhaustive examples versus generated data. Justify keeping simple, hard-coded examples when they already enumerate every interesting state, rather than replacing them with a property and randomized inputs that assert nothing more.
- Limits of hand randomization. Explain why hand-written randomization is easy to introduce but becomes less suitable as a testing practice matures, and why the author favors libraries despite their drawbacks.
- Contextual test strategy. Explain why neither a standard testing pyramid nor a prescribed number of tests can determine the right strategy for every application or feature.
- Unit and integration test tradeoffs. Distinguish what unit tests and integration tests can establish, and explain the limitation of each.
- Limits of code coverage. Explain why a high code-coverage percentage does not necessarily indicate valuable tests or reliable software.
- Investment in test data. Describe why the section favors making data easy to create and making tests consume that data over trying to raise code coverage.
- Narrative comments in tests. Explain why tests often need comments and how useful comments guide a reader through a test’s purpose rather than merely describing its code.
- Modeling away unnecessary tests. Explain why encoding invariants in types can simplify a test suite: states the program cannot express do not need tests.
- Data-oriented testing approach. Explain the section’s closing principle: put what you know into tests, drive tests with data, and let the computer do most of the work.
What it builds on
- Functions, Purity, and Determinism (chapter 6) · Introduces pure and deterministic functions.
- Implementing the non-deterministic shell (chapter 6) · Separates the non-deterministic shell from deterministic work.
- Designing objects with functional core; imperative shell (chapter 10) · Introduces the functional-core and imperative-shell division.
- Programming with immutable data (chapter 2) · Establishes programming with immutable data and values.
- Associativity (chapter 7) · Introduces associativity as an algebraic property.
- Enforce invariants during construction (chapter 3) · Enforcing invariants through the design of types.
Section by section
Introduction
The chapter introduces data-oriented testing and emphasizes that how tests represent data affects their clarity.
- Representation in test code: Explain why representation matters in tests: even small changes to how test data is represented can substantially affect how understandable a test is.
Why it matters: The way a test represents its data can make the test easier to understand.
Key terms: representation, understandability
11.1 A few definitions
This section defines how the book uses the terms unit test, integration test, and mock.
- Unit and integration test boundary: Distinguish unit tests from integration tests using the book’s definitions: unit tests run in isolation, with dependencies mocked if needed; integration tests involve dependencies and IO. Explain why the size or role of a test does not determine its category.
Why it matters: These broad definitions make it clear which kind of test the book means without arguing over a test’s size or name. - Broad meaning of mock: Explain what the book calls a mock, including the kinds of control over an object or function that the term covers.
Why it matters: This definition clarifies what it means when the book says a unit test’s dependencies are mocked.
Key terms: Unit Tests, Integration Tests, isolation, dependencies, IO, Mock, stub, shim, proxy
11.2 Data orientation makes tests better
This section shows how data-oriented code makes tests simpler and more flexible by separating behavior from dependencies and treating test inputs and outputs as data.
- Input and output testing: Explain why pure, deterministic functions are straightforward to test: a test supplies data, calls the function, and asserts on the returned data.
Why it matters: This makes the behavior under test visible in one place. - Dependency setup and test brittleness: Distinguish tests of isolated behavior from tests of code that mixes business logic with dependencies and side-effects.
Why it matters: A passing test is less reassuring when it depends on a complicated simulation of the environment. - Isolated CSV parsing tests: Explain how splitting the FileRepository into a functional core and imperative shell lets a test check CSV deserialization without dealing with the CloudStore.
Why it matters: The repository example shows that an API can have side-effects while its interesting data transformation remains testable in isolation. - Generated data and deep comparison: Describe how generating valid Account states and CSV from them strengthens the parsing test, and what its final assertion checks.
Why it matters: Generated inputs let essentially the same input-and-output test check more states and catch discrepancies in individual fields.
Key terms: pure, deterministic functions, inputs and outputs, non-determinism, mocking, side-effects, brittleness, FileRepository, functional core, imperative shell, CSV
11.3 Writing data-oriented tests
The section explains what good tests communicate and how brittle, opaque, narrowly focused tests fall short.
- Tests as domain explanations: Explain why a good test suite does more than verify behavior and guard against regressions: it shows how the code interacts with the world and which invariants should hold.
Why it matters: A test that explains expected behavior can help future readers understand the software, not just detect a failure. - Signs of ineffective tests: Describe why tests that are brittle, hard to interpret, or narrowly tied to one code path are weak even if they satisfy a coverage metric.
Why it matters: Recognizing these weaknesses is a first step toward improving a test rather than trusting its coverage number.
Key terms: test suite, domain, invariants, regression, bad tests, code path, coverage metric
11.3.1 Use data to show what’s under test
The section shows how to make a test explain its expected result by putting relevant input data and assertions together.
- Input and assertion relationship: Explain why a test should show the relationship between its inputs and assertions, rather than leaving the reader to investigate an unexplained expected value.
Why it matters: Showing where an expected result comes from makes a failing test easier to diagnose. - Locally visible test state: Describe why relevant test state belongs in the test body and how factory functions can keep that state readable.
Why it matters: A reader should not have to search shared helpers to find the facts that explain a test. - Threshold boundary in test data: Explain how including an account exactly at the threshold makes the example test communicate the service’s rule.
Why it matters: A boundary example lets the test explain a domain rule that a vague result count would leave unclear.
Key terms: causal link, inputs, assertions, data, shared setup methods, local reasoning, factory functions, test data, Account, threshold
11.3.2 Data makes tests flexible
The section shows how data-driven tests can vary inputs, expose weaknesses in assertions, and generate many scenarios automatically.
- Flexible threshold-driven test inputs: Explain how deriving a test’s inputs from a threshold makes it easy to try different values and uncover assumptions tied to hard-coded constants.
Why it matters: Varying the data can reveal dependencies that a fixed test value hides. - Weakness of count assertions: Explain why checking only that two sets have the same size can miss an account-filtering bug.
Why it matters: A passing count check can conceal a test that processed the wrong accounts. - Generated subset test scenarios: Describe how computing the power set of test data lets a test exercise every subset, including the empty collection, instead of selecting variations by hand.
Why it matters: Generated subsets let the computer explore combinations a test author might overlook.
Key terms: threshold, hard-coded constants, data oriented tests, sets, comparison operator, Discounts, shouldBeProcessed, power set, subsets, data-oriented tests
11.3.3 Model side-effects with data
This section explains how data-driven tests can describe and verify side-effects even when a method returns nothing useful.
- Data-driven side-effect verification: Explain how a test can use data to specify an expected side-effect and then assert that it happened, without making mocking the focus of the test.
Why it matters: A method does not need to return useful data for its test to remain data-driven.
Key terms: side-effects, data, mocking
11.3.4 Don’t be afraid to introduce new data types
This section argues that tests can use data types to describe meaningful system states and expected behavior, even when the code under test consists entirely of side effects.
- Domain states as test data: Explain how introducing data types can make tests describe the meaningful states and lifecycles of a system, using the report-mirroring service as an example.
Why it matters: Describing states as data makes the test express the domain rather than merely the path taken through the code. - Domain meaning over branch coverage: Distinguish tests that describe what the domain should do from tests that merely exercise code branches and check mock interactions.
Why it matters: A test should give confidence in the behavior’s meaning, not just confirm that a path ran.
Key terms: data modeling, data types, lifecycles, state of the system, branch coverage, domain, semantic level
11.4 Avoiding common problems in test suites
This section shows how hard-coded shared test data creates hidden coupling and argues for keeping useful helpers while removing the hard coding.
- Hidden shared-data coupling: Trace how a test that reuses a helper for one needed Account value can silently depend on other hard-coded values outside the test, and explain why changes to shared data can eventually break many tests.
Why it matters: A test can look simple and reusable while depending on values its author never intended to test. - Shared helpers without hard coding: Explain why the section rejects a blanket ban on shared test data and instead proposes keeping helpful shared functions while removing the hard-coded data that creates dangerous coupling.
Why it matters: The aim is to avoid hidden dependencies without giving up the convenience of shared helpers.
Key terms: coupling, hard coded data, helper functions, shared test data, shared helper functions, hard coding, test data
11.4.1 Randomize unused inputs to prevent coupling
This section explains how randomizing unspecified test inputs prevents hidden coupling and tests the assumption that those inputs do not affect the asserted behavior.
- Random inputs and test coupling: Explain why test-data helpers should randomize attributes a test does not need instead of supplying hard-coded values.
Why it matters: It helps keep tests from silently depending on shared helper data. - Specializing shared test data: Describe how the section recommends setting specific attributes when starting from randomized test data.
Why it matters: Tests can share a randomized starting point while clearly stating the values each test requires. - Assertions across unspecified inputs: Explain what randomizing an unspecified attribute says about the test’s assertion and why that matters as software changes.
Why it matters: Varying supposedly irrelevant inputs can expose subtle interactions and future regressions.
Key terms: helper functions, hard coded values, randomized test data, coupling, Withers, overloaded functions, randomized data, randomizing data, assertions, all possible values
11.5 Using randomized data in tests
This section explains why randomized test data can explore program states that example-based tests often miss.
- Untested Program States: Explain why a test built around one chosen example may leave many possible program states untested.
Why it matters: Seeing the states beyond a chosen example explains why broader test data is useful. - Randomized Testing Perspective: Explain how randomized inputs change what a test tries to establish compared with a hard-coded example, and state the limit of that claim.
Why it matters: This shift helps tests give confidence beyond the particular examples a programmer thought to write.
Key terms: data type, possible values, possible states, example, randomized data, hard-coded state, algebraic style of reasoning, randomizer
11.5.1 Streams + randomization = infinite test data
The section shows how randomized generators and Java streams can produce varied test data, including combined cases, empty collections, and nested inputs.
- Stream-Based Random Test Data: Explain how Stream.generate turns a function that produces one randomized value into a source of arbitrary amounts of test data, including collections tailored to a test.
Why it matters: One generator can supply many examples without manually constructing each input. - Combined and Partitioned Test Cases: Describe the two ways the section organizes generated cases: combining specialized generators into one stream, or generating a broad set and dividing it into interesting subsets.
Why it matters: Both approaches let tests work with a broad range of cases while still focusing on cases of interest. - Random-Sized and Nested Inputs: Explain why tests can randomize the amount of generated data, including zero, and nest generators to create varied collections of collections.
Why it matters: Varying sizes and nesting generators expose input combinations a fixed-size example may miss.
Key terms: Stream.generate, randomized data, specialized collections, generators, Stream.concat, interesting subsets, for all, randomize the amount of data, edge cases, Lists of Lists of Accounts
11.5.2 Exercise all the states in your program
This section shows how to use randomized test data to exercise the states relevant to a test’s invariant, rather than relying on a few hand-picked examples.
- State Coverage Through Randomization: Explain why tests should be judged by the states their assertions exercise, not just by whether a few chosen examples pass. Describe how randomized thresholds broaden the test’s intent while a fixed region can still leave important states untested.
Why it matters: Looking for untested states makes a test’s claim stronger than adding more fixed examples of the same case. - Invariant Guided Data Generation: Explain how to steer random generators toward states needed for an invariant without specifying every input. Use the account’s relationship to a threshold to illustrate which spend values matter.
Why it matters: Guidance makes generated data likely to exercise the cases the assertion depends on without reducing the test to fixed examples. - Two State Generation Patterns: Distinguish building generators for specific subsets and combining them from generating broadly and extracting interesting subsets. Explain how the latter ensures an equality case in the threshold example.
Why it matters: These alternatives let a test author choose the approach that makes the important cases and invariant clearest.
Key terms: states, randomized data, threshold, invariant, generators, spend(), moving from specific to general, moving from general to specific, concat, equality case
11.5.3 One test, multiple roles
Generated data lets one test serve different purposes by changing how many inputs it runs against.
- Generated input scale and test roles: Explain how changing the amount of generated data lets the same test serve as a unit test or a load test, and how larger runs can reveal a service’s limits.
Why it matters: You can explore how a system behaves at different scales without writing a separate test for each scale.
Key terms: generating data, unit test, load test, generators
11.5.4 The most useful tests are often the simplest
This section explains how simple randomized round-trip tests can expose data-handling problems at system boundaries.
- Randomized Boundary Round Trips: Explain how repeatedly round-tripping randomized data through boundaries such as a REST API or database can reveal serialization problems, including problems in dependencies, before users encounter them.
Why it matters: A small round-trip test can catch boundary and library failures before they affect users.
Key terms: randomized test data, round-trip, system boundaries, serialization
11.5.5 Verifying algebraic properties.
This section explains how randomized examples can give confidence in algebraic properties when a data type has too many possible states to test exhaustively.
- Randomized checks of algebraic laws: Explain how to test an algebraic law when inputs such as Strings or BigDecimals make exhaustive testing impractical, and distinguish growing confidence from proof.
Why it matters: You can test laws over very large input spaces without mistaking a large sample for exhaustive proof.
Key terms: algebraic laws, state space, randomizer, String, BigDecimal
11.5.6 Randomization is just another tool
This section explains when randomized testing helps, when deterministic examples are better, and how test failures can guide the choice.
- Randomization and determinism tradeoff: Explain why randomized test data is not suitable for every test: it trades away some determinism, while hard-coded data and mocks are often used precisely to make tests behave predictably.
Why it matters: It helps you choose randomization for a reason rather than treating it as an automatic improvement. - Exhaustive examples versus generated data: Justify keeping simple, hard-coded examples when they already enumerate every interesting state, rather than replacing them with a property and randomized inputs that assert nothing more.
Why it matters: It keeps a test's complexity proportional to what it actually needs to check. - Test outcomes as feedback: Describe how to use bugs that escape a test suite to reconsider the testing approach, including whether a missing case or a generator that missed an assumed case explains the gap.
Why it matters: Investigating how a bug escaped can improve the suite, not just fix the immediate defect.
Key terms: randomized test data, determinism, mock, hard code data, interesting states, algebraic property, randomized data, generative power, bugs sneaking through your suite, generator
11.5.7 Favor using libraries when generating data
This section explains why hand-written randomization is a useful starting point for tests but libraries are preferable as randomized testing grows more sophisticated.
- Limits of hand randomization: Explain why hand-written randomization is easy to introduce but becomes less suitable as a testing practice matures, and why the author favors libraries despite their drawbacks.
Why it matters: The choice of testing tools affects whether randomized tests remain useful as the suite grows. - Replaying randomized failures: Explain why a failing randomized test needs a recorded seed that can be fed back into the suite, and why always using that seed defeats the point of fresh randomization.
Why it matters: Replaying failures makes them investigable without giving up new inputs on ordinary runs. - Input selection and diagnosis: Describe the choices involved in generating random test values and the debugging problem created by many generated examples.
Why it matters: A randomized test needs both useful inputs and a way to identify what caused a failure.
Key terms: randomization, libraries, testing style, repeatable randomization, seed, distribution, edge cases, debuggability
11.6 A few closing thoughts on testing
This section argues that testing strategies should fit the software and explains the complementary strengths and limits of unit and integration tests.
- Contextual test strategy: Explain why neither a standard testing pyramid nor a prescribed number of tests can determine the right strategy for every application or feature.
Why it matters: It helps you choose tests for the risks in your software instead of following a template. - Unit and integration test tradeoffs: Distinguish what unit tests and integration tests can establish, and explain the limitation of each.
Why it matters: Knowing what each test type can and cannot show helps you pick the right tool for a particular question.
Key terms: testing pyramid, testing “shape”, unit tests, integration tests, mocks, outside world
11.6.1 Data coverage is more important than test coverage
This section argues that useful test data and meaningful assertions matter more than a code-coverage percentage.
- Limits of code coverage: Explain why a high code-coverage percentage does not necessarily indicate valuable tests or reliable software.
Why it matters: A coverage number can look reassuring even when the tests give little confidence. - Investment in test data: Describe why the section favors making data easy to create and making tests consume that data over trying to raise code coverage.
Why it matters: Easier-to-create test data lets you improve tests without chasing a metric that may not reflect their value.
Key terms: coverage, domain and its properties, confidence, data coverage, consumers of data
11.6.2 Embrace the humble comment
This section explains why narrative comments make tests easier to understand by revealing the purpose and hidden context that test code alone may not show.
- Narrative comments in tests: Explain why tests often need comments and how useful comments guide a reader through a test’s purpose rather than merely describing its code.
Why it matters: A short explanation of why a test exists can spare the next reader from having to reconstruct its hidden context.
Key terms: narrative, comments, institutional knowledge
11.6.3 The best tests are the ones you don’t have to write
Better modeling can eliminate tests by making invalid states impossible to express.
- Modeling away unnecessary tests: Explain why encoding invariants in types can simplify a test suite: states the program cannot express do not need tests.
Why it matters: A stronger model can reduce the number of tests you need to write and maintain.
Key terms: invariants, type system, NonEmpty, NonNegative
11.7 Wrapping up
The section closes by summarizing data-oriented testing as putting what you know into tests, driving them with data, and letting the computer do much of the work.
- Data-oriented testing approach: Explain the section’s closing principle: put what you know into tests, drive tests with data, and let the computer do most of the work.
Why it matters: This is the section’s concise takeaway about how to approach testing.
Key terms: data-oriented testing, data
11.8 Summary
Where this is used later
- Writing Integration Tests (chapter 12) · The discussion of writing integration tests uses the book’s definition of an integration test.
- Drive your integration tests with data (chapter 12) · Using data to show what a test is about
- Embrace concurrency, beat up your software, and find bugs (chapter 12) · The comparison to randomized tests, which also explore cases not specified one by one.
- Impose algebraic properties on your dependencies (chapter 12) · Provides the testing rationale for checking algebraic properties of dependencies.
- Building a reliable integration test suite (chapter 12) · Frames the goal of building a reliable integration test suite as more than reaching a coverage threshold.
Concepts explained in this chapter
- randomizationRandomization is the generation or selection of varying test inputs to exercise a broader range of program states than fixed examples.
- software testingTesting is the practice of executing checks that assess whether software behaves according to specified expectations.
- test data generationA test data generator is a mechanism that produces input values or cases for testing software according to defined generation rules.
- test suiteA test suite is a collection of automated or executable tests that assess software against specified expectations.
- state spaceA state space is the set of all possible combinations of values representing the states of a domain.
- randomized test dataRandomized test data is test input whose values or cases are generated with deliberate variation rather than fixed in advance.
- mockingA mock is a test-controlled replacement or implementation that stands in for an object or function and allows its behavior or interactions to be controlled.
- dataData is information represented independently of the behaviors that operate on it.
- integration testIntegration testing evaluates interactions between multiple software components or external dependencies.
- unit testA unit test is an automated check that evaluates a relatively small, isolated unit of software behavior.
- data-oriented programmingData-oriented programming is a programming approach that organizes software around explicit data representations and operations on those representations.
- couplingA dependency relationship in which one software component or test relies on the implementation details or behavior of another.
- hardcodingHard coding is embedding environment-specific values or infrastructure assumptions directly in application code.
- assertionAn assertion is an executable check that a specified condition holds for a value, state, or computation.
- factoryA factory function is a function that constructs and returns an instance or structured test datum through a defined parameter interface.
- determinismA deterministic function is a computation whose output is completely determined by its inputs and remains consistent for repeated evaluations with those inputs.
- functional coreThe deterministic core is the central part of an application containing domain logic whose results do not depend on side effects or nondeterministic interactions.
- imperative shellAn architectural pattern that isolates deterministic computation in a functional core and places side effects and nondeterministic interactions in an imperative shell.
- pure functionA pure function is a computation that produces the same output for the same inputs without observable side effects or dependence on mutable external state.
- code coverageCode coverage is a metric indicating which portions of source code are exercised by tests.
- side effectA side effect is an observable change or interaction caused by a computation beyond producing its returned result.
- data modelingData modeling is the design process of identifying domain information, its meaning, and its representation.
- data typeA data type is a classification of data that determines its representation, permitted values, and valid operations.
- lifecycleA lifecycle is the sequence of states and changes an entity undergoes over the course of its existence.
- stateThe data values that describe the current condition of an object or entity.
- invariantAn invariant is a condition that must remain true across the valid states of a data structure or computation.
- accuracyCorrectness is the property that an implementation produces the behavior and results specified by its intended semantics and requirements.
- functionA computation that maps input values to an output value or values without requiring mutation of the input values.
- illegal stateAn illegal state is a program state that violates the validity constraints imposed by its data model or domain rules.
- algebraic reasoningAlgebraic thinking is the practice of reasoning about software operations through equations and properties that characterize their behavior.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
