Replication maintains copies of topic-partition records across Kafka brokers. It preserves data when a disk, machine, or broker fails, while the broker stores records as raw bytes in partition logs. section 2.1 section 2.4
Each topic partition has one leader and one or more followers. The leader handles produce and consume requests; followers fetch the leader’s records and append them to their own logs. A follower whose data and offsets match the leader is an in-sync replica (ISR). section 2.9 section 2.9.1
The replication lifecycle is:
- A producer sends records to the partition leader, which appends them to its log. section 2.9.1
- Followers consume the new records from the leader and append them to their logs. section 2.9.1
- Followers that catch up become ISRs; followers that exceed replica.lag.time.max.ms are removed from the ISR list. section 2.9.1
- An ISR can be elected leader if the current leader fails or becomes unavailable. section 2.9.1
Replication factor of three
A replication factor of three creates one leader and two followers for each topic partition. The leader may briefly contain records that followers have not yet copied; once caught up, all three replicas are in sync. section 2.9.1
Figure 2.13 The Kafka replication process
Trace the broker-to-broker replication flow from the partition leader to its followers. section 2.9.1
