A Kafka log segment is a discrete, file-backed portion of a topic partition. The broker appends records to the active segment, then creates a new active segment when the configured size or age limit is reached. Older segments remain available for consumers. section 2.6
Each segment works with companion indexes to support storage and reads:
- The .log file stores records and associated fields such as offset, size, and physical position. section 2.6.3
- The .index file maps relative offsets to physical positions, letting the broker binary-search for a read starting point. section 2.6.3
- The .timeindex file maps timestamps to relative offsets. section 2.6.3
- Segment filenames use the base offset, while index offsets are stored relative to that base offset. section 2.6.3
Rolling a purchases partition
A purchases-0 directory may begin with 00000000000000000000.log, .index, and .timeindex. After the active segment rolls, another group such as 00000000000000037348.log and its indexes appears. The broker can then remove an entire older segment when its records satisfy time- or size-based retention. section 2.6.3 section 2.6.1
Figure 2.6 Creating new segments
Visualizes how incoming records move from the active segment into successive segments. section 2.6
