Chapter 18: Dress rehearsal patterns for building confidence
The introduction explains why teams should regularly run scenarios against real services as well as stubs, while recognizing the feedback and reliability challenges this creates.
After this chapter you will be able to
- Continuous integration dress rehearsals. Explain what integration testing checks and why scenarios should regularly run against actual services rather than relying only on stubs.
- Database Baseline Choices. Explain why scenarios need an effectively identical starting database state, and distinguish clean-start, seeded, and snapshot baselines.
- Concurrent Scenario Isolation. Explain how concurrent use of a shared database can make scenarios interfere, and compare the isolation strategies and their limits.
- Repeatable Scenario Starting States. Explain why serial execution alone does not ensure repeatability and compare recreating, resetting, and partitioning the database.
- Database Execution Cost Trade-offs. Explain where real-database scenario execution costs arise, how baseline size affects setup and maintenance, and what to do before optimizing.
- Reset Before Scenario Execution. Describe how the Reset Before pattern prepares a clean, known database state for each scenario and how the team initially applies it.
- Advantages Over Reset After. Justify resetting before rather than after a scenario in terms of interrupted runs and failure diagnosis.
- Scenario-owned database data. Explain that Clean Start begins every scenario with an empty database and makes the scenario create the records it needs. Describe how this upfront work lets scenarios be maintained independently rather than depend on shared baseline data.
- Fast generic database emptying. Describe the recommended way to implement the empty start: use fast, generic database-level emptying, commonly TRUNCATE TABLE, rather than recreating the database or deleting specific test records. Explain the MySQL example's handling of foreign key checks.
- Shared baseline test data. Explain how the Seeded Data pattern extends Clean Start by adding commonly needed records after a database reset, reducing repeated scenario setup. Describe why WIMP's order scenarios need menu records even when the particular menu item does not matter.
- Custom menu scenario isolation. Describe how scenarios requiring specific menu items can keep their explicit Given step while replacing seeded menu records, and explain when excluding a scenario from seeding may be preferable.
- Conditional database reset. Explain how the Skippable Reset pattern uses changes made during one scenario to decide which reset work is needed before the next.
- Reliable modification detection. Justify why change detection must cover modifications made by automation, the application, or the database itself before a reset can safely be skipped.
- When snapshot databases are necessary. Explain why scenarios may need an existing snapshot database instead of test-first database management, particularly for legacy applications or systems that require a very large record set to start.
- Fragility of snapshot test data. Explain how refreshing a snapshot can break scenarios and why lack of test data ownership increases maintenance and constrains new scenarios.
- Change journal reset sequence. Trace how the Undo Changes pattern records scenario changes and restores the baseline before the next scenario.
- Baseline Entity Lookup. Explain how Entity Finder uses existing entities in a snapshot baseline instead of creating all the prerequisite data for a scenario, particularly when setup is complicated or time-consuming.
- Reliable Lookup and Fallback. Describe why entity searches must check all relevant suitability criteria and what the automation can do when no suitable entity is found.
What it builds on
- Stub Dependency pattern: stubbing an external service (chapter 15) · The use of stubs for dependencies during scenario automation.
- Isolated scenarios (chapter 12) · Scenarios should be isolated.
- (chapter intro) (chapter 14) · The need to manage input test data for scenarios.
- Evolving with fast feedback (chapter 12) · The value of fast feedback from scenario automation.
- Hooks (chapter 12) · The use of hooks in scenario automation
- Local Scenario Context pattern: learning to remember (chapter 13) · Scenario context and the isolation concern associated with static fields.
Section by section
Introduction
The introduction explains why teams should regularly run scenarios against real services as well as stubs, while recognizing the feedback and reliability challenges this creates.
- Continuous integration dress rehearsals: Explain what integration testing checks and why scenarios should regularly run against actual services rather than relying only on stubs.
Why it matters: Regular real-service runs can reveal problems that fast stub-based checks miss. - Real-service testing tradeoffs: Describe the constraints the introduction identifies for trustworthy integration-test feedback: slower runs and services whose datasets the team cannot readily control.
Why it matters: A useful integration check must give results the team can trust without making feedback impractically slow.
Key terms: integration testing, stubs, dress rehearsals, continuous integration, speed of feedback, test results, external datasets, external, shared resources
18.1 Challenges of using real databases for test execution
Explains how baseline data, test-data choices, isolation, repeatability, and reset costs shape an automation strategy that uses a real database.
- Database Baseline Choices: Explain why scenarios need an effectively identical starting database state, and distinguish clean-start, seeded, and snapshot baselines.
Why it matters: The starting data determines both how reliable scenarios are and how much setup each one needs. - Scenario Test Data Approaches: Distinguish inserting, querying, and hardcoding scenario data, including when each is feasible and why their resilience differs.
Why it matters: The way a scenario obtains its data affects whether baseline changes break the automation. - Concurrent Scenario Isolation: Explain how concurrent use of a shared database can make scenarios interfere, and compare the isolation strategies and their limits.
Why it matters: Preventing scenarios from changing each other’s data avoids flaky results without overlooking the cost of slower execution. - Repeatable Scenario Starting States: Explain why serial execution alone does not ensure repeatability and compare recreating, resetting, and partitioning the database.
Why it matters: A test suite can be flaky even when it runs one scenario at a time if earlier runs leave data behind. - Database Execution Cost Trade-offs: Explain where real-database scenario execution costs arise, how baseline size affects setup and maintenance, and what to do before optimizing.
Why it matters: Measuring the actual bottleneck helps keep feedback fast without making baseline data unnecessarily fragile.
Key terms: baseline, clean start, seeded, snapshot, insert, query, hardcode, test data management, isolation, serial execution
18.2 Reset Before pattern: reset database before scenario execution
Explains how resetting persistent test data before each scenario makes executions repeatable while leaving failed-run data available for diagnosis.
- Persistent Data Between Scenarios: Explain why disabling parallel execution does not prevent scenarios using a real database from affecting one another.
Why it matters: This is the problem a database reset must solve, even when scenarios run one at a time. - Reset Before Scenario Execution: Describe how the Reset Before pattern prepares a clean, known database state for each scenario and how the team initially applies it.
Why it matters: A known starting state makes scenario results more isolated and repeatable. - Advantages Over Reset After: Justify resetting before rather than after a scenario in terms of interrupted runs and failure diagnosis.
Why it matters: Reset timing affects both the reliability of subsequent runs and the evidence available when a run fails.
Key terms: real database, repeatability, scenario executions, Reset Before pattern, BeforeScenario hook, database reset, clean, known state, after-scenario hook, residual data, debugging
18.3 Clean Start pattern: start each scenario with an empty database
This section explains how the Clean Start pattern gives each database-backed scenario an empty starting point and why fast, generic database emptying is preferable to recreating the database.
- Scenario-owned database data: Explain that Clean Start begins every scenario with an empty database and makes the scenario create the records it needs. Describe how this upfront work lets scenarios be maintained independently rather than depend on shared baseline data.
Why it matters: A scenario that supplies its own data is less vulnerable to unrelated changes in other scenarios' test data. - Fast generic database emptying: Describe the recommended way to implement the empty start: use fast, generic database-level emptying, commonly TRUNCATE TABLE, rather than recreating the database or deleting specific test records. Explain the MySQL example's handling of foreign key checks.
Why it matters: A quick, record-independent reset keeps the clean start practical as the test suite grows. - Alternative reset pitfalls: Distinguish the cautions given for batch deletes, application-level deletes, transaction-wrapped scenarios, and dropping and recreating the database.
Why it matters: Knowing these pitfalls helps a team choose an emptying approach that is reliable for its application and database.
Key terms: Clean Start pattern, blank slate, baseline data, avalanche-effect, TRUNCATE TABLE, foreign key checks, ResetDatabase, batch delete, delete-triggers, foreign key constraints
18.4 Seeded Data pattern: share common data across scenario executions
This section explains how carefully chosen seed data can simplify scenarios that use a real database while preserving a way for scenarios with different data requirements to set up their own context.
- Shared baseline test data: Explain how the Seeded Data pattern extends Clean Start by adding commonly needed records after a database reset, reducing repeated scenario setup. Describe why WIMP's order scenarios need menu records even when the particular menu item does not matter.
Why it matters: A shared baseline removes repetitive setup without abandoning a predictable starting state. - Custom menu scenario isolation: Describe how scenarios requiring specific menu items can keep their explicit Given step while replacing seeded menu records, and explain when excluding a scenario from seeding may be preferable.
Why it matters: Explicit scenario requirements stay readable and reliable even when the shared baseline changes. - Careful seed data selection: Justify choosing only frequently needed seed records rather than an arbitrary test database or production snapshot, and describe why the existing data access layer is preferred for inserting them.
Why it matters: A small, maintainable baseline is less likely to interfere with scenarios or break as the database evolves.
Key terms: Seeded Data pattern, Clean Start pattern, seed data, foreign key constraint, Given step, step definition, @no-menu-seed-data, data access layer, low-level SQL
18.5 Skippable Reset pattern: do not reset database after read-only scenario executions
This section explains how the Skippable Reset pattern speeds scenario execution by omitting database reset work when reliable change detection shows it is unnecessary.
- Conditional database reset: Explain how the Skippable Reset pattern uses changes made during one scenario to decide which reset work is needed before the next.
Why it matters: Skipping unnecessary reset work can provide faster feedback without changing what scenarios test. - Reliable modification detection: Justify why change detection must cover modifications made by automation, the application, or the database itself before a reset can safely be skipped.
Why it matters: A faster suite is not worthwhile if skipped resets make later scenario results unreliable. - Change tracking options: Describe the change-detection options presented and the conditions that make them useful or risky.
Why it matters: The available way to detect changes depends on how the application and its database record modifications. - Cross-scenario tracking state: Explain why Skippable Reset needs information preserved between scenarios and how that creates an isolation risk.
Why it matters: Remembering changes between scenarios must not make the scenarios themselves depend on one another.
Key terms: Skippable Reset pattern, baseline, partial database reset, re-seeding, change detection, read-only, false test errors, timestamps, data access library events, log events
18.6 Patterns for snapshot databases
This section explains why some BDD scenarios must use snapshot databases despite the maintenance problems those datasets create.
- When snapshot databases are necessary: Explain why scenarios may need an existing snapshot database instead of test-first database management, particularly for legacy applications or systems that require a very large record set to start.
Why it matters: It explains why a team might use snapshots even when it prefers test-first automation. - Fragility of snapshot test data: Explain how refreshing a snapshot can break scenarios and why lack of test data ownership increases maintenance and constrains new scenarios.
Why it matters: It shows why relying on a prepared dataset can limit both reliable execution and the scenarios a team chooses to write.
Key terms: test-first approach, snapshot databases, legacy applications, hardcoded entities, test data ownership
18.6.1 Undo Changes pattern: only reset modified data
This section explains when to use the Undo Changes pattern and how to restore a database baseline by recording and reversing scenario changes.
- Undo Changes use case: Explain why tracking and undoing scenario changes can be preferable to rebuilding baseline data.
Why it matters: It helps you choose a reset approach that fits the baseline you actually have. - Change journal reset sequence: Trace how the Undo Changes pattern records scenario changes and restores the baseline before the next scenario.
Why it matters: The journal is what lets the next scenario start from the original baseline without rebuilding it. - Change tracking approaches: Describe the change-tracking options and the limitations or simplifications identified for each.
Why it matters: The way changes are captured must match how the application actually writes data. - Referential integrity during reversal: Explain how reversing database changes can preserve referential integrity.
Why it matters: A rollback that ignores relationships between records may fail even when every individual change was logged.
Key terms: Undo Changes pattern, baseline data, Seeded Data, Skippable Reset, change journal, reset database before-scenario hook, reverse operation, baseline, data access library, ORM tool
18.6.2 Entity Finder pattern: find suitable data instead of creating it
This section explains how the Entity Finder pattern uses existing snapshot-baseline data to set up scenarios, and why that shortcut needs careful safeguards.
- Baseline Entity Lookup: Explain how Entity Finder uses existing entities in a snapshot baseline instead of creating all the prerequisite data for a scenario, particularly when setup is complicated or time-consuming.
Why it matters: It can make scenario automation practical when creating all the required data would take substantial work. - Reliable Lookup and Fallback: Describe why entity searches must check all relevant suitability criteria and what the automation can do when no suitable entity is found.
Why it matters: A lookup is useful only if it finds genuinely suitable data and behaves predictably when it cannot. - Baseline Dependency Tradeoffs: Justify when Entity Finder is worth using despite its risks, including its possible role as a temporary solution.
Why it matters: It helps a team treat database lookup as a deliberate tradeoff rather than a universally safe shortcut.
Key terms: Entity Finder pattern, snapshot baseline, Given step definitions, prerequisite entities, search criteria, baseline database versions, backup plan, automated entity creation, flaky tests, baseline database
18.7 Summary
Where this is used later
- (chapter intro) (chapter 21) · Challenges of testing with shared external resources.
- Resource Pooling pattern: leasing pooled resources to scenarios running in parallel (chapter 21) · Resource pooling develops the separate-instance isolation strategy.
- Test Environment Initialization pattern: getting ready for test execution (chapter 21) · The earlier Clean Start database reset used a before-scenario hook.
Concepts explained in this chapter
- clean start patternA pattern in which each scenario starts with an empty service or database and creates the data it needs.
- skippable reset patternA pattern that skips a reset when the preceding scenario made no changes requiring one.
- test data managementThe practice of defining and maintaining the data provided to the system under test in software testing.
- restore before patternA pattern that resets state to a known baseline before each scenario, regardless of how the previous scenario ended.
- seeded data patternA pattern that starts scenarios with a selected initial dataset to simplify setup or improve execution speed.
- entity finder patternA pattern that finds entities in existing data that satisfy a scenario’s prerequisites instead of creating them.
- undo changes patternA pattern that records changes made during a scenario and reverses them to restore the baseline.
- seed dataCarefully selected data pre-populated in a database before scenarios execute.
- exclusive data partitionA partitioning strategy that prevents changes made by one scenario from being visible to other running scenarios.
- insert approachA test data management approach in which automation code creates the entities a scenario needs on demand.
- query approachA test data management approach that queries existing data to find suitable entities for tests.
- seeded baselineA test data management approach in which every scenario starts with a standard set of data in the database.
- sequential processingExecuting scenarios sequentially rather than concurrently against a service.
- resource poolingA pattern that assigns reusable instances of a shared resource for exclusive use while work executes.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
