liveBook · Manning

Data temperature

As explained in Cloud Observability in Action

Data temperature is a classification of data based on how frequently and how quickly it must be accessed.

Data temperature classifies logs by how recently and how frequently they are accessed, guiding retention, query speed, and storage cost. section 2.3.3

For centralized logs, the practical categories are:

  • Hot data is the active troubleshooting or development working set. Keep it readily accessible and fast to query, typically for days to weeks. section 2.3.3
  • Warm data may be needed later but does not require immediate access. Retention can extend from months to years. section 2.3.3
  • Cold data is rarely used, often retained for regulatory reasons, and belongs in cheap, durable object storage despite access delays of minutes or hours. section 2.3.3
Table 2.1 Overview of the costs of logs (view table figure)

 

Overhead

Considerations

Instrumentation

Small and simple to add a log line

Widely adopted and supported by programming languages

Agent

Some runtime, lots of options, and few standards for formats (syslog, OTLP)

A runtime footprint, ingestion performance, reliability, and security

Destination

Long-term storage can be expensive.

Data temperature (access recency or frequency) can guide retention

Use the cost overview to compare instrumentation, agent, and destination considerations when assigning retention by data temperature. section 2.3.3

How Cloud Observability in Action covers it

How deep it goes
Defines hot, warm, and cold categories with access expectations, retention ranges, and storage trade-offs.
Examples
Uses centralized logs, Amazon S3 Glacier, and destinations such as Splunk, CloudWatch, OpenSearch, and Datadog.
What it assumes
Assumes familiarity with logs and centralized log destinations while defining temperature categories from the ground up.
What you can do after
Operate/configure log retention by assigning data to fast, delayed, or archival storage according to access patterns.
How it connects
Connects log retention with troubleshooting, storage cost, query accessibility, regulatory retention, and object storage.

This explainer was generated by Manning's knowledge graph from the text of Cloud Observability in Action. The section links point to the parts of the book it draws on.