liveBook · Manning

Is/is not analysis

As explained in Build High Value Software Systems

A problem-analysis technique that compares where, when, and under what conditions a problem occurs with conditions where it does not.

An intermittent failure is easier to investigate when you stop calling it “random.” Is/Is Not analysis compares conditions where a problem occurs with closely related conditions where it does not. The contrast narrows the search to a meaningful boundary: a region, platform, user group, environment, data characteristic, or point in time. It is especially useful when production fails but local testing does not. section 8.2.4

Table 8.2 Is and Is Not for identifying the gaps
Dimension
Is
Is Not
Environment
Production
Staging, Local
Region
US-West
US-East, EU
Platform
Mobile
Desktop
Timing
After Thursday's deploy
Before Thursday

The checkout comparison makes four boundaries visible at once: production rather than staging or local, US-West rather than other regions, mobile rather than desktop, and after Thursday’s deploy rather than before it. section 8.2.4

From boundaries to a candidate cause

Suppose checkout crashes only for mobile users in US-West after Thursday’s deploy. Those boundaries point toward a feature flag enabled for that group at that time. The flag activates a checkout flow that skips cart validation, allowing empty carts to crash order creation. The table directs the investigation; checking the flagged flow establishes the cause. A restart or a vague label such as “intermittent” would leave the conditions unexplained. section 8.2.4 section 8.0

Use the comparison as a debugging aid:

  • Observe the system first: compare failing and successful endpoints, users, regions, and time windows using logs and other operational evidence. section 8.3
  • Start with environment, timing, user segment, platform, and data characteristics. Add dimensions if the first pass does not narrow the field. section 8.2.4
  • Investigate the boundary rather than treating it as proof. Use 5 Whys to trace its cause, a fishbone diagram to check for missed factors, and Pareto analysis to prioritize by impact. section 8.2.5

How Build High Value Software Systems covers it

How deep it goes
Shows a comparison table and incident investigations, then hands deeper cause tracing to other root cause techniques.
Examples
Uses a checkout comparison across environments, regions, platforms, and deployment timing, plus an email-delivery incident.
What it assumes
Assumes familiarity with deployments and production debugging; teaches the comparison table directly.
What you can do after
Diagnose/debug with it by narrowing an inconsistent failure to testable conditions before tracing its underlying cause.
How it connects
Connects observability and root cause analysis with 5 Whys, fishbone diagrams, and Pareto analysis.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build High Value Software Systems. The section links point to the parts of the book it draws on.