Hot data

As explained in Kafka for Architects

Hot data is recently generated or frequently accessed data that benefits from low-latency storage.

Hot data is recently generated or frequently accessed data that benefits from low-latency storage. In tiered storage, it remains on Kafka broker local disks while older data moves to cheaper remote storage. section 7.1.5

This arrangement balances access speed, retention, and cost:

• Keep frequently accessed data local for low-latency reads. section 7.1.5
• Move older, rolled-over segments and metadata to Amazon S3, Azure Blob Storage, Google Cloud Storage, or another remote system. section 7.1.5
• Let consumers use normal operations without knowing whether requested data is local or remote. section 7.1.5

Figure 7.13 When the tiered storage feature is enabled, the cold data is moved periodically to cheaper, cloud-based storage.

Shows hot data remaining on broker disks while cold data is periodically offloaded to cheaper cloud storage. section 7.1.5

How Kafka for Architects covers it

How deep it goes

Explains storage placement, consumer transparency, configuration properties, and remote-storage failure handling.

Examples

Uses broker local disks for hot data and cloud storage for older segments in a tiered-storage architecture.

What you can do after

Operate/configure tiered storage with cluster- or topic-level remote-storage settings while preserving ordinary consumer access.

How it connects

Connects hot data with cold data, tiered storage, local disks, remote storage, Kafka topics, and consumers.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Related concepts

• datakind of · Data is represented information, observations, measurements, or other values that can be stored, processed, transmitted, analyzed, or interpreted.
• cachinguses · Caching is the practice of storing reusable data or computation results so later access can avoid repeating retrieval or computation.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.