Data tiering
Data tiering is the placement of data across storage classes with differing cost, capacity, performance, or accessibility characteristics, according to access patterns or policies.
Kafka for Architects
Tiered storage is a storage arrangement that places data on different storage tiers according to factors such as age, speed, and cost.
- How it is used
- Optimizes Kafka retention by keeping frequently accessed events local and moving older events to cheaper remote storage.
- How deep it goes
- Explains placement, configuration, consumer behavior, cleanup safeguards, and remote failures, but not implementation internals.
- Examples
- Uses broker-local disks, Amazon S3, Azure Blob Storage, Google Cloud Storage, and a hot-versus-cold storage diagram.
- What you can do after
- Operate/configure Kafka tiering using cluster-level or topic-level remote-storage settings and account for cold-data availability.
Apache Kafka in Action
Tiered storage is a storage strategy that places data on different storage tiers according to factors such as age, cost, or access frequency.
- How it is used
- Uses Kafka tiered storage to retain accessible recent data while controlling the cost of long-term historical retention.
- How deep it goes
- Explains the retention split and retrieval complication but does not provide configuration steps or implementation internals.
- Examples
- Uses recent Kafka data and offloaded historical events to illustrate age-based access and retention-cost trade-offs.
- What you can do after
- Evaluate retention designs by balancing access performance, historical availability, retrieval complexity, and storage cost.
Related concepts
- Apache Kafkaused by · Apache Kafka is a distributed event-streaming platform for publishing, storing, processing, and consuming streams of records.
- Kafka brokerimplemented by · A Kafka broker is a server process in an Apache Kafka cluster that stores records and handles producer, consumer, and other client requests.
- remote log managerimplemented by · A remote log manager is a Kafka component that manages remote log segments, metadata, and transfers between local brokers and remote storage.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.