A Kafka partition can grow far beyond a practical single file. Kafka divides its partition log into segments: each segment’s .log file holds a portion of the messages, with companion index files that help locate them. The files are named for the segment’s first offset. section 8.3.4 section 8.3.3
When a broker accepts a produced message, an I/O thread appends it to the current segment for that partition. The write reaches the operating system’s page cache; Kafka normally leaves the later disk flush to the operating system. Replication, rather than frequent forced flushes, is the recommended route to reliability here. section 8.2.1
$ kafka-dump-log.sh \
--files ~/kafka/data/kafka1/\
products.prices.changelog.file-structure-0/0[…]000.log
Dumping ~/kafka/data/kafka1/
products.prices.changelog.file-structure-0/0[…]000.log
Log starting offset: 0
baseOffset: 0 lastOffset: 1 count: 1 […] position: 0
CreateTime: 1738455138490 size: 68 […] crc: 1092464612
[…]
Inspect a .log file with kafka-dump-log.sh --files. The output identifies the segment’s starting offset and shows each batch’s offsets, message count, size and byte position. section 8.3.3
Kafka rolls to a new segment when a size or age limit is reached—by default, 1 GiB or seven days. A topic can override these limits with segment.bytes and segment.ms. Lowering segment.ms alone does not immediately create a file: produce another message to trigger the check. section 8.3.4
$ kafka-configs.sh \
--alter \
--topic products.prices.changelog.file-structure \
--add-config segment.ms=60000 \
--bootstrap-server localhost:9092
This kafka-configs.sh command sets the topic’s segment.ms to 60,000 milliseconds. After the interval, produce messages and inspect the partition directory to check for a new segment. section 8.3.4
$ ls -1 ~/kafka/data/kafka1/products.prices.changelog.file-structure-0 00000000000000000000.index 00000000000000000000.log 00000000000000000000.timeindex 00000000000000000003.index 00000000000000000003.log 00000000000000000003.snapshot 00000000000000000003.timeindex leader-epoch-checkpoint
The directory listing shows two .log files and their companion index files. Their names begin with different offsets, marking the starts of successive segments. section 8.3.4
Rolling also matters for cleanup. Log compaction ignores the current segment because rewriting a segment that is still receiving messages could cause inconsistencies. On busy brokers, segments have another cost: every segment is kept as an open file, so the available file descriptors matter. section 10.4.1 section 6.3.1

