An unknown unknown is a relevant condition, behavior, or failure mode whose possibility has not previously been recognized. Production systems can fail in ways teams never anticipated, including interactions among familiar components that behave unexpectedly. section 5.1.3
Telemetry helps move these problems toward the known-knowns category. Tracing and transaction IDs reveal how information flows between systems, while flexible time-series analysis helps investigators recognize anomalies that fixed dashboards may miss. section 5.1.3 section 5.5
Unexpected production behavior
A pod may reschedule in an unanticipated situation, or an API may respond unpredictably when it receives malformed data. Overlapping time-series data across systems can help a human investigator find the anomaly and its root cause. section 5.1.3
Address unknown unknowns by evolving observability practices:
- Instrument software and systems so processes can be followed across states and service boundaries. section 5.1.3
- Combine broad, high-quality observability data with trend, outlier, or predictive analysis to expose weaknesses that were not previously recognized. section 5.1.3
- Make workloads observable by default and support investigation through self-service platform capabilities. section 5.5
Figure 5.3 Knowing what to look for when querying observability data is an evolving process.
Use the figure to follow how knowing what to query in observability data changes as unfamiliar failure patterns become recognized. section 5.1.3
