liveBook · Manning

Log segment

As explained in Apache Kafka in Action

A log segment is a file containing a contiguous portion of the records in a Kafka partition log.

A Kafka partition can grow far beyond a practical single file. Kafka divides its partition log into segments: each segment’s .log file holds a portion of the messages, with companion index files that help locate them. The files are named for the segment’s first offset. section 8.3.4 section 8.3.3

When a broker accepts a produced message, an I/O thread appends it to the current segment for that partition. The write reaches the operating system’s page cache; Kafka normally leaves the later disk flush to the operating system. Replication, rather than frequent forced flushes, is the recommended route to reliability here. section 8.2.1

$ kafka-dump-log.sh \
    --files ~/kafka/data/kafka1/\
products.prices.changelog.file-structure-0/0[…]000.log
Dumping ~/kafka/data/kafka1/
products.prices.changelog.file-structure-0/0[…]000.log
Log starting offset: 0
baseOffset: 0 lastOffset: 1 count: 1 […] position: 0
CreateTime: 1738455138490 size:  68 […] crc: 1092464612
[…]

Inspect a .log file with kafka-dump-log.sh --files. The output identifies the segment’s starting offset and shows each batch’s offsets, message count, size and byte position. section 8.3.3

Kafka rolls to a new segment when a size or age limit is reached—by default, 1 GiB or seven days. A topic can override these limits with segment.bytes and segment.ms. Lowering segment.ms alone does not immediately create a file: produce another message to trigger the check. section 8.3.4

$ kafka-configs.sh \
    	--alter \
    --topic products.prices.changelog.file-structure \
    --add-config segment.ms=60000 \
    --bootstrap-server localhost:9092

This kafka-configs.sh command sets the topic’s segment.ms to 60,000 milliseconds. After the interval, produce messages and inspect the partition directory to check for a new segment. section 8.3.4

$ ls -1 ~/kafka/data/kafka1/products.prices.changelog.file-structure-0
00000000000000000000.index
00000000000000000000.log
00000000000000000000.timeindex
00000000000000000003.index
00000000000000000003.log
00000000000000000003.snapshot
00000000000000000003.timeindex
leader-epoch-checkpoint

The directory listing shows two .log files and their companion index files. Their names begin with different offsets, marking the starts of successive segments. section 8.3.4

Rolling also matters for cleanup. Log compaction ignores the current segment because rewriting a segment that is still receiving messages could cause inconsistencies. On busy brokers, segments have another cost: every segment is kept as an open file, so the available file descriptors matter. section 10.4.1 section 6.3.1

How Apache Kafka in Action covers it

How deep it goes
Explains segment files, offset-based names, inspection and rolling controls; defers the fuller treatment of cleanup.
Examples
Uses kafka-dump-log.sh output, a segment.ms configuration command and a partition-directory listing with successive .log files.
What it assumes
Introduces segments through the topic-and-partition hierarchy while assuming familiarity with Kafka command-line tools and produced messages.
What you can do after
Operate/configure it by inspecting segment files, adjusting topic rolling limits and producing a message to trigger a roll check.
How it connects
Connects segments to Kafka partitions, offset indices, page cache, log compaction and broker file descriptors.

Also explained in

Compare how 2 books explain log segment →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.