An intermittent failure is easier to investigate when you stop calling it “random.” Is/Is Not analysis compares conditions where a problem occurs with closely related conditions where it does not. The contrast narrows the search to a meaningful boundary: a region, platform, user group, environment, data characteristic, or point in time. It is especially useful when production fails but local testing does not. section 8.2.4
Table 8.2 Is and Is Not for identifying the gaps
|
Dimension
|
Is
|
Is Not
|
|
Environment
|
Production
|
Staging, Local
|
|
Region
|
US-West
|
US-East, EU
|
|
Platform
|
Mobile
|
Desktop
|
|
Timing
|
After Thursday's deploy
|
Before Thursday
|
The checkout comparison makes four boundaries visible at once: production rather than staging or local, US-West rather than other regions, mobile rather than desktop, and after Thursday’s deploy rather than before it. section 8.2.4
From boundaries to a candidate cause
Suppose checkout crashes only for mobile users in US-West after Thursday’s deploy. Those boundaries point toward a feature flag enabled for that group at that time. The flag activates a checkout flow that skips cart validation, allowing empty carts to crash order creation. The table directs the investigation; checking the flagged flow establishes the cause. A restart or a vague label such as “intermittent” would leave the conditions unexplained. section 8.2.4 section 8.0
Use the comparison as a debugging aid:
- Observe the system first: compare failing and successful endpoints, users, regions, and time windows using logs and other operational evidence. section 8.3
- Start with environment, timing, user segment, platform, and data characteristics. Add dimensions if the first pass does not narrow the field. section 8.2.4
- Investigate the boundary rather than treating it as proof. Use 5 Whys to trace its cause, a fishbone diagram to check for missed factors, and Pareto analysis to prioritize by impact. section 8.2.5
