A broker or disk can fail without warning. Kafka limits the damage by keeping copies of each topic partition on multiple brokers. Unlike a backup that must be restored, these copies remain on running brokers so another broker can take over when one fails. section 4.2.3 section 4.5
Set a replication factor for each topic, usually when creating it. The factor cannot exceed the number of available brokers: three brokers can hold three replicas of a partition, but not four. A cluster with at least three brokers is a practical starting point because taking one offline for maintenance still leaves room for another failure. section 4.2.3 section 4.3.2
$ kafka-topics.sh \
--create \
--topic products.prices.replication \
--partitions 3 \
--replication-factor 3 \
--bootstrap-server localhost:9092
Created topic products.prices.changelog.replication.
Create a topic with three partitions and a replication factor of three on a three-broker cluster. section 4.2.3
Each partition has one leader replica. Producers and consumers normally communicate with that leader; followers fetch new messages from it and can take over if it fails. Replicas can briefly differ when a follower falls behind. section 4.2.3 section 8.5
$ kafka-topics.sh \
--describe \
--topic products.prices.replication \
--bootstrap-server localhost:9092
Topic: products.prices.replication PartitionCount: 3 ReplicationFactor: 3
Configs: #1
Topic: products.prices.replication Partition: 0 Leader: 2 Replicas: 2,1,3
Isr: 2,1,3 Elr: #2
Topic: products.prices.replication Partition: 1 Leader: 3 Replicas: 3,2,1
Isr: 3,2,1 Elr:
Topic: products.prices.replication Partition: 2 Leader: 1 Replicas: 1,3,2
Isr: 1,3,2 Elr:
Inspect the topic’s partitions, leaders, replica assignments, and in-sync replicas (Isr). Different partitions can have different leaders. section 4.2.3
An in-sync replica (ISR) has received the leader’s messages within a specified time frame. The high watermark marks the offset replicated and committed by all ISRs, affecting which messages consumers can read. If replication lags, consumption or production can slow or stop. Settings such as replica.lag.time.max.ms and acks=all therefore need balancing against performance. section 8.5 section 8.4
Replication improves durability and availability, but it is not a backup service: Kafka does not provide the cold backups described here. Its reliance on copies across brokers also assumes those brokers do not all fail and lose their data at once. section 4.2.3 section 6.3
