Chapter 19: Technical note patterns for quick error recovery
The introduction explains why failing BDD scenarios need early, context-rich error reporting to make failures in a CI/CD pipeline quicker to diagnose.
After this chapter you will be able to
- Symptoms versus failure causes. Explain why the visible failure of a BDD scenario may not reveal what went wrong earlier in its execution.
- Early context-rich error reporting. Justify reporting errors close to their cause and preserving relevant execution details, especially for failures seen through a remote CI/CD pipeline.
- Diagnostic assertion messages. Explain why the Descriptive Assertion pattern puts relevant context directly in an assertion error message. Use the order example to describe what a useful message adds beyond expected and actual numbers.
- Failure message verification. Describe when to check an assertion's error message and how to check it again after changing the assertion.
- Supplementary context for failures. Explain why a supplementary attachment is useful when diagnostic information is non-textual or too large for an assertion error message, using an application log or UI screenshot as an example.
- Failure attachment workflow. Describe how an after-scenario hook can save supplementary information, optionally restrict attachment creation to failed scenarios, and register the saved file with the test result.
- Diagnostic purpose of test logs. Explain why a scenario failure needs an execution log as well as an error message, and how the log complements a concise scenario.
- Test action boundary logging. Describe what to record at the boundary between the test and automation layers and why a shared Test Action entry point makes that practical.
- Hierarchical execution log structure. Describe how logging scopes make the relationship between test actions and their lower-level operations visible.
- Unified scenario diagnostic record. Explain why the Embedded Application Log pattern puts application messages in the test log rather than leaving them in a separate attachment, and how hierarchical logging can present those messages.
- Adjustable application log detail. Explain how to balance readable everyday test logs with extra detail for diagnosing difficult failures, including the retry example.
- Behavioral success versus application health. Explain why a BDD scenario can pass its behavioral checks while the application is behaving unhealthily, and why the Corroboration Check pattern adds health verification.
- Health check timing and placement. Describe the two ways the section proposes checking application health, including when an after-scenario check should run.
- Fail-fast setup verification. Explain when to add a fail-fast check to a context-setting step and why it should run before subsequent steps.
What it builds on
- (chapter intro) (chapter 12) · Assumes the role of automated BDD scenarios in providing feedback.
- Scenario granularity (chapter 6) · Assumes that a scenario focuses on illustrating behavior.
- Understandable diagnostics (chapter 12) · The need for understandable diagnostics in scenario automation.
- Hooks (chapter 12) · The concept of scenario automation hooks.
- Keep your scenarios BRIEF (chapter 7) · The principle of keeping scenarios BRIEF.
- Test Action pattern: simplified management of test actions (chapter 15) · The Test Action pattern's shared action infrastructure.
- Scenario Resource pattern: managing test resources (chapter 16) · Access to information about the executing scenario, including its tags.
- Attempt Action pattern: when failure is allowed (chapter 15) · negative scenarios using the Attempt Action pattern
Section by section
Introduction
The introduction explains why failing BDD scenarios need early, context-rich error reporting to make failures in a CI/CD pipeline quicker to diagnose.
- Regression feedback in CI/CD: Explain how existing BDD scenarios serve as regression tests and why a failure in the CI/CD pipeline demands prompt attention.
Why it matters: A blocked pipeline makes the usefulness of failure feedback immediately important. - Symptoms versus failure causes: Explain why the visible failure of a BDD scenario may not reveal what went wrong earlier in its execution.
Why it matters: Recognizing a symptom as distinct from its cause helps the team investigate the whole execution rather than only the failing step. - Early context-rich error reporting: Justify reporting errors close to their cause and preserving relevant execution details, especially for failures seen through a remote CI/CD pipeline.
Why it matters: Useful details captured during execution can shorten the time a failing scenario blocks delivery.
Key terms: regression tests, CI/CD pipeline, BDD scenario, root cause, regression problem, early error detection, runtime information
19.1 Descriptive Assertion pattern: including context information in the error message
This section shows how to make assertion failures easier to diagnose by putting relevant context in their error messages and checking those messages during development.
- Diagnostic assertion messages: Explain why the Descriptive Assertion pattern puts relevant context directly in an assertion error message. Use the order example to describe what a useful message adds beyond expected and actual numbers.
Why it matters: A useful failure message can shorten the investigation that follows a failed pipeline scenario. - Failure message verification: Describe when to check an assertion's error message and how to check it again after changing the assertion.
Why it matters: Checking the message when a failure is available avoids having to reconstruct the assertion's context later. - Assertion library choices: Describe how an assertion library may help produce understandable expectations and errors, and why a team should use an agreed assertion style consistently.
Why it matters: A helpful library and a consistent style make assertions easier to read and maintain. - Accessible scenario reports: Explain how the section contrasts build-server assertion output with reports that business stakeholders can access, and what Cucumber Messages changes about report formatting.
Why it matters: Accessible, consistent reports make scenario results easier to share beyond the delivery team.
Key terms: Descriptive Assertion pattern, assertion error message, context information, Red-Green-Refactor loop, failing scenario, assertion, assertion library, fluent API, AwesomeAssertions, Cucumber Messages
19.2 Supplementary Attachment pattern: attaching execution context information to the test result
This section explains when to attach execution context to a scenario result and how to preserve it for diagnosing failures.
- Supplementary context for failures: Explain why a supplementary attachment is useful when diagnostic information is non-textual or too large for an assertion error message, using an application log or UI screenshot as an example.
Why it matters: The attachment gives someone investigating a failure useful context without overloading the error message. - Failure attachment workflow: Describe how an after-scenario hook can save supplementary information, optionally restrict attachment creation to failed scenarios, and register the saved file with the test result.
Why it matters: This workflow makes failure evidence available in the test result while avoiding the usual cost of generating it for every run.
Key terms: Supplementary Attachment pattern, execution context, error message, application log, after-scenario hook, scenario execution status, test result attachment, AddAttachment
19.3 Test Log pattern: recording scenario execution details
The section explains how the Test Log pattern records scenario execution details across automation layers so failures are easier to diagnose without cluttering scenarios.
- Diagnostic purpose of test logs: Explain why a scenario failure needs an execution log as well as an error message, and how the log complements a concise scenario.
Why it matters: A useful failure report shows the path to the error, not just its final symptom. - Test action boundary logging: Describe what to record at the boundary between the test and automation layers and why a shared Test Action entry point makes that practical.
Why it matters: One shared boundary can capture the actions most useful for reconstructing an execution. - Selective diagnostic log detail: Explain how additional test-layer and low-level infrastructure details can be recorded without making normal test output too verbose.
Why it matters: Filtering preserves useful routine output while leaving deeper evidence available for difficult failures. - Hierarchical execution log structure: Describe how logging scopes make the relationship between test actions and their lower-level operations visible.
Why it matters: Nesting makes it easier to see which operation belongs to which test action when tracing a failure. - Caution with AI log analysis: Explain the value and limitation of using AI to analyze execution logs after a test failure.
Why it matters: A plausible diagnosis or a passing test is not proof that the underlying failure was fixed.
Key terms: Test Log pattern, BRIEF principle, scenario execution errors, test layer, automation layer, Test Action pattern, TestAction base class, input and output parameters, LogDebug, debug
19.4 Embedded Application Log pattern: including the application log with the test log
This section explains how embedding application logs in the test log creates one diagnostic record, and how to control its verbosity so that record remains useful.
- Unified scenario diagnostic record: Explain why the Embedded Application Log pattern puts application messages in the test log rather than leaving them in a separate attachment, and how hierarchical logging can present those messages.
Why it matters: One log makes it easier to connect a scenario’s actions with what the application recorded. - Application log integration options: Describe how application messages can be routed into the test log and why the configuration varies between projects.
Why it matters: The routing approach has to fit the application’s logging and test setup. - Adjustable application log detail: Explain how to balance readable everyday test logs with extra detail for diagnosing difficult failures, including the retry example.
Why it matters: Selective detail preserves a readable everyday log while making elusive failures easier to investigate.
Key terms: Embedded Application Log pattern, test log, application log, hierarchical log messages, test logger, application’s logging output, in-process, background job, default verbosity level, log category
19.5 Corroboration Check pattern: monitoring the application’s health indicators
This section explains how the Corroboration Check pattern catches unexpected application health issues that ordinary scenario assertions can miss.
- Behavioral success versus application health: Explain why a BDD scenario can pass its behavioral checks while the application is behaving unhealthily, and why the Corroboration Check pattern adds health verification.
Why it matters: A passing scenario is more trustworthy when it has not overlooked signs of hidden trouble. - Health check timing and placement: Describe the two ways the section proposes checking application health, including when an after-scenario check should run.
Why it matters: Checking at the right point preserves useful functional failure information while still detecting health issues. - Expected health issue exceptions: Distinguish unexpected health issues from warnings or errors that a scenario legitimately produces, and describe how the section handles those exceptions.
Why it matters: Treating expected or unrelated warnings as failures would make the health check misleading.
Key terms: Corroboration Check pattern, health indicators, functional checks, SUT, Test Action, Attempt Action, after-scenario hook, application log, false negatives, negative scenarios
19.6 Fail-fast Check pattern: ensure that the preconditions have been met
This section explains how fail-fast checks verify scenario preconditions immediately so setup errors are easier to diagnose.
- Fail-fast setup verification: Explain when to add a fail-fast check to a context-setting step and why it should run before subsequent steps.
Why it matters: An immediate setup failure is easier to fix than a misleading failure later in the scenario. - Asynchronous precondition readiness: Trace how the sales-record example detects that asynchronous context setup has actually finished before the When step proceeds.
Why it matters: A completed call is not always a completed precondition when work continues in the background. - Generic and custom checks: Distinguish when a reusable fail-fast check is preferable and when a problematic setup call needs its own check.
Why it matters: A reusable check catches the same kind of silent setup failure across calls, while a specific call may still need its own verification.
Key terms: Fail-fast Check pattern, context, precondition, Given, background job, When step, timeout, generic fail-fast check, custom fail-fast check, RestApiContext
19.7 Summary
Concepts explained in this chapter
- test log patternA test-automation pattern that records structured execution details across automation layers to support failure diagnosis.
- embedded application log patternA logging pattern that includes selected application log messages within a test log to provide a single source of diagnostic information.
- descriptive assertionA pattern that includes diagnostic context in assertion failure messages.
- supplementary attachment patternA scenario-automation pattern that attaches relevant context to an assertion error message when it cannot be included in the message itself.
- corroboration check patternA scenario-automation pattern that checks application logs for entries indicating potential issues, including issues that have not yet caused behavioral failures.
- sanity checkA pattern that verifies setup preconditions immediately, before later steps depend on them.
- structured loggingLogging that organizes entries to convey meaningful structure, such as relationships between nested execution steps.
- test result attachmentA file or other item registered with a test result so it can be inspected alongside the test outcome.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
