Data loss is the unintended destruction or unavailability of stored data. In Kafka, it can result from weak acknowledgment settings, insufficient in-sync replicas, or storage failures. section 5.1.2 section 16.1.3
Reduce data-loss risk by combining replication controls with storage safeguards:
- Set a reasonable min.insync.replicas value; with replication factor 3, a recommended starting point is 2. section 5.1.2
- Do not assume acks=all prevents loss when only one replica remains in sync; in that case, it behaves like acks=1. section 5.1.2
- Monitor disk capacity and stop Kafka before volumes become full, especially when the stored data is important. section 16.1.3
- Replace failed disks and allow the broker to replicate their data from the respective leaders until it rejoins the in-sync replicas. section 16.1.3
Testing insufficient replicas
Create a topic with replication factor 3 and min.insync.replicas=2, stop two brokers, and inspect the topic. Only one ISR remains; producing with acks=all then fails with NOT_ENOUGH_REPLICAS instead of accepting the message. section 5.1.2
$ kafka-topics.sh \
--create \
--topic products.prices.changelog.min-isr-2 \
--replication-factor 3 \
--partitions 3 \
--config min.insync.replicas=2 \
--bootstrap-server localhost:9092
Use the topic-creation command to configure three replicas and require two in-sync replicas before accepting writes. section 5.1.2
