A remote log manager handles Kafka’s remote log segments, associated metadata, and transfers between brokers and remote storage. It enables a hot-and-cold architecture: active data remains local, while older data moves to cheaper storage. section 7.1.5
Key operational consequences include:
- Moves closed segments and their index files after segment roll; uploads are scheduled automatically rather than manually. section 7.1.5
- Keeps local logs until remote copies complete, preserving consistency even after local retention limits are reached. section 7.1.5
- Requires followers to copy local data and maintain remote log metadata for complete replica functionality. section 7.1.5
- Uses segment.bytes, segment.ms, and remote.log.manager.task.interval.ms to influence cadence, with remote-storage quotas limiting infrastructure load. section 7.1.5
Hot and cold Kafka data
A deployment retains actively accessed data and recovery or backfill windows on hot storage, then offloads older closed segments to cheaper cloud-based storage. Figure 7.13 depicts this periodic movement. section 7.1.5
