Chapter 11: Chapter 11. Testing and debugging multithreaded applications
The chapter introduces the difficulty of testing and debugging concurrent code and outlines an issues-first approach to finding and removing bugs.
After this chapter you will be able to
- Concurrency bug categories. State that concurrent code can have ordinary bugs, but bugs directly related to concurrency typically fall into two broad categories: unwanted blocking and race conditions.
- Unwanted blocking and dependencies. Explain that a thread is blocked when it cannot proceed because it is waiting, and that waiting becomes unwanted when other threads need that thread to act before they can proceed.
- Deadlock and livelock differences. Distinguish deadlock's mutually dependent blocking waits from livelock's active checking loops, and describe their effects on progress and CPU usage.
- Race Conditions and Data Races. Distinguish a race condition, where behavior depends on the relative scheduling of threads, from a data race, where unsynchronized concurrent access to shared memory causes undefined behavior. Explain that some race conditions are benign, while data races commonly arise from incorrect atomic synchronization or missing mutex protection.
- Broken Shared Data Invariants. Describe how races can break value-based or temporal invariants: a thread may observe partial updates, use data another thread deleted, or perform an operation that was supposed to happen in a particular order.
- Thread and Data Lifetimes. Explain how a thread can access data after that data has been destroyed, especially when it retains a reference to a local variable that goes out of scope. State that thread and data lifetimes must be coordinated, and that a manual join() must not be skippable by an exception.
- Thorough concurrency code review. Explain why locating bugs by reading code requires a careful review of what the code actually does, rather than a quick check for obvious problems or coding-standard violations.
- Testing after code review. Explain why thorough review does not replace testing: bugs may still have been missed, and testing helps confirm that multithreaded code works.
- Concurrent code review questions. Describe the questions a reviewer should ask about shared data protection, where other threads might be, which mutexes they might hold, required ordering between operations, and whether loaded data remains valid.
- Assumed interference investigation. Explain how assuming another thread can modify data helps trace a possible bug through corner cases and interleavings, especially when different mutexes protect data over its lifetime or other code can obtain a pointer or reference to it.
- Mutex release data changes. Explain why shared data must be treated as potentially changed after a mutex is released and reacquired, using separate stack top() and pop() calls as an example; describe combining the operations under one lock as the remedy.
- Scheduling and intermittent failures. Explain why identical inputs can produce different results in multithreaded tests: thread scheduling can change between runs, so a race condition may cause only occasional failures.
- Focused tests and concurrency diagnosis. Describe how to test the smallest code unit that could reveal a problem, such as testing a concurrent queue directly, and how running with one thread helps distinguish an ordinary bug from a concurrency-related one.
- Concurrent queue test scenarios. Describe how queue tests should vary both the operations running together and the queue’s starting state, including empty, full, and partially full cases.
- Separation of concurrent boundaries. Describe how to separate interthread communication and shared-data access from logic that operates within one thread, so ordinary tests cover the latter and multithreaded tests focus on the smaller concurrent boundary. Apply this split to the section's state-machine or read/transform/update examples.
- Stress testing and its limits. Explain how brute-force testing repeatedly runs code, possibly with many threads, to increase the chance of encountering a problematic schedule. Describe why fine-grained tests, test-harness design, and representative hardware and operating systems matter to the confidence those runs provide.
- Exhaustive combination simulation. Describe how combination simulation testing records threads’ data accesses, locks, and atomic operations, then applies C++ memory-model rules to explore permitted combinations and detect problems. Explain why its exhaustive checking is chiefly practical for small, fine-grained tests.
- Specialized testing libraries. Explain how special threading or synchronization libraries can reveal problems during test runs by checking access to mutex-protected data, recording inconsistent lock orders, or giving a test control over lock acquisition and condition-variable notification. Distinguish these checks and controlled scenarios from exhaustive simulation.
- Four phases of a test. Describe the four parts of a multithreaded test: general setup, per-thread setup, code intended to run concurrently, and checks after the threads finish. Explain why expensive preparation should happen before the concurrent part.
- Ready and go coordination. Trace how each task signals its own readiness, waits on a shared go future, and then runs its test operation after the main thread has received every ready signal and issued go. Explain why simply starting two threads does not reliably make their operations overlap.
- Exception safe test completion. Explain how the test waits for async tasks and retrieves the popped value, and why it releases the go signal if an exception interrupts setup or execution. Describe how the task futures' destructors help ensure the threads finish.
- Performance gains and scalability. Explain that concurrency used as an optimization must be measured, and that scalability means performance should improve as more cores become available, rather than improving only on a small machine.
- Core configurations and contention. Describe testing on different processor configurations to reveal the scalability graph, including at least a single-processor system and one with as many cores as available. Explain that contention for a data structure can make code that scales at low core counts perform badly at higher counts.
What it builds on
- Deadlock: the problem and a solution (chapter 3) · The earlier explanation of deadlock.
- Race conditions (chapter 3) · The earlier introduction to race conditions.
- Objects, memory locations, and concurrency (chapter 5) · The relationship between shared memory locations and concurrent access.
- Spotting race conditions inherent in interfaces (chapter 3) · The risks of operations on shared data whose interfaces do not preserve invariants.
- Waiting in exceptional circumstances (chapter 2) · Waiting for a thread to finish even in exceptional circumstances.
- A thread-safe queue using fine-grained locks and condition variables (chapter 6) · The thread-safe queue example with separate head and tail mutexes.
- A thread-safe stack using locks (chapter 6) · The thread-safe stack design that combines access and removal.
- Synchronizing operations and enforcing ordering (chapter 5) · The C++ memory-model rules governing synchronization and ordering.
Section by section
Introduction
The chapter introduces the difficulty of testing and debugging concurrent code and outlines an issues-first approach to finding and removing bugs.
- Testing and debugging roles: Distinguish testing, which seeks bugs in concurrent code, from debugging, which removes them, and explain that neither has an easy solution.
Why it matters: Keeping the two roles distinct helps you choose techniques for finding bugs and then fixing them. - Issues-first chapter approach: Describe why the chapter looks at problems that can arise before presenting ways to locate bugs, design multithreaded tests, and test performance.
Why it matters: Recognizing the possible problems gives the later testing techniques a purpose.
Key terms: testing, debugging, concurrent code, concurrency-related bugs, code review, multithreaded tests, performance
11.1 Types of concurrency-related bugs
Introduces unwanted blocking and race conditions as the two broad categories of bugs directly related to concurrency.
- Concurrency bug categories: State that concurrent code can have ordinary bugs, but bugs directly related to concurrency typically fall into two broad categories: unwanted blocking and race conditions.
Why it matters: These categories organize the chapter’s discussion of concurrency-related bugs.
Key terms: unwanted blocking, race conditions, concurrency-related bugs
11.1.1 Unwanted blocking
This section explains when waiting becomes an unwanted blocking problem and distinguishes deadlock, livelock, and waits for external input.
- Unwanted blocking and dependencies: Explain that a thread is blocked when it cannot proceed because it is waiting, and that waiting becomes unwanted when other threads need that thread to act before they can proceed.
Why it matters: This distinction helps you identify a harmful wait rather than treating every wait as a bug. - Deadlock and livelock differences: Distinguish deadlock's mutually dependent blocking waits from livelock's active checking loops, and describe their effects on progress and CPU usage.
Why it matters: The two failures can look alike to a user but call for different clues when debugging. - External input blocking risk: Explain why a thread that performs tasks needed by other threads should not block waiting for I/O or other external input that might never arrive.
Why it matters: An indefinite external wait can hold up work elsewhere in the application.
Key terms: blocked, unwanted blocking, deadlock, livelock, spin lock, I/O, external input
11.1.2 Race conditions
The section distinguishes harmless scheduling-dependent behavior from harmful race conditions, describes three common kinds of race-related bugs, and explains why some are hard to locate.
- Race Conditions and Data Races: Distinguish a race condition, where behavior depends on the relative scheduling of threads, from a data race, where unsynchronized concurrent access to shared memory causes undefined behavior. Explain that some race conditions are benign, while data races commonly arise from incorrect atomic synchronization or missing mutex protection.
Why it matters: Not every scheduling-dependent outcome is a bug, but a data race is a specific, serious one. - Broken Shared Data Invariants: Describe how races can break value-based or temporal invariants: a thread may observe partial updates, use data another thread deleted, or perform an operation that was supposed to happen in a particular order.
Why it matters: A race can violate assumptions about both the state of data and the order of operations. - Thread and Data Lifetimes: Explain how a thread can access data after that data has been destroyed, especially when it retains a reference to a local variable that goes out of scope. State that thread and data lifetimes must be coordinated, and that a manual join() must not be skippable by an exception.
Why it matters: Correct synchronization does not help if the data a running thread uses no longer exists. - Misleading Symptoms of Races: Explain why deadlock or livelock often looks like a hang that can be investigated through the involved threads and synchronization objects, whereas data races, broken invariants, and lifetime issues may cause crashes or wrong output far from and long after the faulty code.
Why it matters: The place where a program fails may not be the place where a thread caused the damage.
Key terms: race condition, data race, undefined behavior, atomic operations, mutex, broken invariants, temporal, partial updates, double-free, lifetime issues
11.2 Techniques for locating concurrency-related bugs
This section introduces careful code review and multithreaded testing as complementary ways to find concurrency-related bugs.
- Thorough concurrency code review: Explain why locating bugs by reading code requires a careful review of what the code actually does, rather than a quick check for obvious problems or coding-standard violations.
Why it matters: A superficial review can miss bugs even when the code looks familiar. - Testing after code review: Explain why thorough review does not replace testing: bugs may still have been missed, and testing helps confirm that multithreaded code works.
Why it matters: Review and testing provide different opportunities to catch bugs.
Key terms: reviewing code, concurrency issues, nonconcurrency issues, testing multithreaded code, reviewing the code
11.2.1 Reviewing code to locate potential bugs
This section shows how deliberate code review can uncover subtle concurrency bugs by questioning data protection, thread interleavings, and assumptions about shared data.
- Deliberate fresh perspective review: Explain why multithreaded code needs a thorough, considered review and how another reviewer—or a break followed by a detailed self-explanation—can provide a fresh perspective.
Why it matters: Subtle timing bugs are unlikely to reveal themselves in a quick glance. - Concurrent code review questions: Describe the questions a reviewer should ask about shared data protection, where other threads might be, which mutexes they might hold, required ordering between operations, and whether loaded data remains valid.
Why it matters: These questions turn a general review into a check of specific ways threads can interfere. - Assumed interference investigation: Explain how assuming another thread can modify data helps trace a possible bug through corner cases and interleavings, especially when different mutexes protect data over its lifetime or other code can obtain a pointer or reference to it.
Why it matters: Trying to disprove a possible interference bug exposes assumptions that an ordinary line-by-line reading can miss. - Mutex release data changes: Explain why shared data must be treated as potentially changed after a mutex is released and reacquired, using separate stack top() and pop() calls as an example; describe combining the operations under one lock as the remedy.
Why it matters: A lock protects an operation only while it is held, not an assumption carried across separate calls.
Key terms: code review, reviewer, concurrent access, mutexes, ordering requirements, valid, corner case, mutex, public data, pointer
11.2.2 Locating concurrency-related bugs by testing
The section explains why concurrency bugs are hard to reproduce and how focused tests, single-thread comparisons, varied operation scenarios, and test-environment choices help locate them.
- Scheduling and intermittent failures: Explain why identical inputs can produce different results in multithreaded tests: thread scheduling can change between runs, so a race condition may cause only occasional failures.
Why it matters: A passing run does not rule out a concurrency bug. - Focused tests and concurrency diagnosis: Describe how to test the smallest code unit that could reveal a problem, such as testing a concurrent queue directly, and how running with one thread helps distinguish an ordinary bug from a concurrency-related one.
Why it matters: A small test narrows the faulty code, while a single-thread run checks whether concurrency is necessary for the failure. - Concurrent queue test scenarios: Describe how queue tests should vary both the operations running together and the queue’s starting state, including empty, full, and partially full cases.
Why it matters: Different combinations of callers and queue states can expose different concurrency failures. - Thread counts and test conditions: Explain how the number of threads, available processing cores, processor architecture, and scheduling of simultaneous operations affect the design and environment of a concurrent test.
Why it matters: A test must create the conditions under which the suspected interactions can occur.
Key terms: thread scheduling, race condition, indeterminate, concurrent queue, single thread, worker threads, push(), pop(), empty queue, full queue
11.2.3 Designing for testability
Design multithreaded code so most logic can be tested without concurrency, leaving a smaller, more manageable set of thread interactions to test.
- Testability before implementation: Explain why testability deserves particular attention in multithreaded code and describe how to plan tests before writing it: consider inputs, troublesome conditions, and ways to stimulate problematic behavior.
Why it matters: Planning for tests can shape the design before concurrency makes problems harder to isolate. - Separation of concurrent boundaries: Describe how to separate interthread communication and shared-data access from logic that operates within one thread, so ordinary tests cover the latter and multithreaded tests focus on the smaller concurrent boundary. Apply this split to the section's state-machine or read/transform/update examples.
Why it matters: A smaller concurrent boundary makes the hardest tests more tractable. - Hidden shared library state: Explain that library calls may keep internal state that becomes shared when multiple threads call them, even when shared data is not obvious in the application code; describe protecting and synchronizing those calls or using a concurrency-safe alternative.
Why it matters: An apparently isolated operation can still introduce shared state through a library call.
Key terms: designing for testability, test harness, communication paths between threads, state logic, message routing code, read shared data/transform data/update shared data, internal variables, shared data, protection and synchronization
11.2.4 Multithreaded testing techniques
The section compares three ways to expose scheduling-dependent bugs in multithreaded tests: repeated runs, exhaustive simulation, and specialized testing libraries.
- Stress testing and its limits: Explain how brute-force testing repeatedly runs code, possibly with many threads, to increase the chance of encountering a problematic schedule. Describe why fine-grained tests, test-harness design, and representative hardware and operating systems matter to the confidence those runs provide.
Why it matters: A test that passes repeatedly may still never create the conditions that trigger a bug. - Exhaustive combination simulation: Describe how combination simulation testing records threads’ data accesses, locks, and atomic operations, then applies C++ memory-model rules to explore permitted combinations and detect problems. Explain why its exhaustive checking is chiefly practical for small, fine-grained tests.
Why it matters: Simulation can explore permitted interactions that ordinary repeated runs may never encounter, but its cost limits where it is useful. - Specialized testing libraries: Explain how special threading or synchronization libraries can reveal problems during test runs by checking access to mutex-protected data, recording inconsistent lock orders, or giving a test control over lock acquisition and condition-variable notification. Distinguish these checks and controlled scenarios from exhaustive simulation.
Why it matters: Special libraries can detect or deliberately expose concurrency problems without exploring every possible execution.
Key terms: brute-force testing, stress testing, scheduling permutations, false confidence, combination simulation testing, C++ memory model, race conditions, deadlocks, synchronization primitives, mutexes
11.2.5 Structuring multithreaded test code
This section shows how to structure a multithreaded test so its operations have a good chance of running concurrently and its results can be checked safely.
- Four phases of a test: Describe the four parts of a multithreaded test: general setup, per-thread setup, code intended to run concurrently, and checks after the threads finish. Explain why expensive preparation should happen before the concurrent part.
Why it matters: Separating preparation from the operations under test makes the test more likely to exercise the intended interaction. - Ready and go coordination: Trace how each task signals its own readiness, waits on a shared go future, and then runs its test operation after the main thread has received every ready signal and issued go. Explain why simply starting two threads does not reliably make their operations overlap.
Why it matters: Thread startup time can otherwise let one operation finish before the other thread begins. - Queue outcome assertions: Explain what the push/pop queue test checks after completion, distinguishing the permitted results for blocking and nonblocking pop().
Why it matters: A concurrent test must allow results permitted by the interface while rejecting inconsistent final states. - Exception safe test completion: Explain how the test waits for async tasks and retrieves the popped value, and why it releases the go signal if an exception interrupts setup or execution. Describe how the task futures' destructors help ensure the threads finish.
Why it matters: A test harness must not leave its workers waiting forever when the test itself throws.
Key terms: general setup, thread-specific setup, concurrent execution, after completion, std::promise, std::shared_future, ready signal, go signal, std::launch::async, blocking pop()
11.2.6 Testing the performance of multithreaded code
This section explains how to test whether multithreaded code improves performance and scales across machines with different numbers of cores.
- Performance gains and scalability: Explain that concurrency used as an optimization must be measured, and that scalability means performance should improve as more cores become available, rather than improving only on a small machine.
Why it matters: A speedup on one machine does not show that the code will scale well. - Serial work speedup limit: Explain that work which runs on only one thread limits the maximum possible speedup, so the code’s design should be examined before testing to set a realistic performance target.
Why it matters: A realistic target helps distinguish a design limit from a disappointing test result. - Core configurations and contention: Describe testing on different processor configurations to reveal the scalability graph, including at least a single-processor system and one with as many cores as available. Explain that contention for a data structure can make code that scales at low core counts perform badly at higher counts.
Why it matters: Testing a range of core counts can expose performance regressions that a small-machine benchmark misses.
Key terms: performance, scalability, serial portion, performance gain, contention, scalability graph, single-processor system
11.3 Summary
The chapter summary recaps the kinds of concurrency bugs covered and the approaches presented for finding and testing them.
Concepts explained in this chapter
- code reviewCode review is the systematic examination of source code to identify defects, improve quality, and check conformance to standards.
- multithreaded testMultithreaded test code coordinates multiple threads so that selected operations execute concurrently and their results can be checked.
- combination simulation testingCombination simulation testing records concurrent operations and systematically explores permitted combinations of those operations to detect concurrency errors.
- stress testBrute-force testing repeatedly executes software, often under varied or high-load conditions, to expose failures that occur only under particular execution schedules.
- concurrent testingTesting software for defects that arise when multiple threads or computations execute concurrently.
- race conditionA race condition is a program condition in which behavior depends on the timing or ordering of operations performed by concurrent threads.
- scalabilityScalability is the ability of a system or algorithm to increase its capacity as available resources increase.
- deadlockDeadlock is a state in which two or more threads are indefinitely blocked, each waiting for a resource or event that another blocked thread must provide.
- livelockA livelock is a state in which threads repeatedly change or retry operations in response to one another without making useful progress.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.