Data consistency means keeping persisted Kafka data exactly as intended across a distributed cluster. It becomes difficult when brokers fail, recover, or fail in succession: producers and consumers must balance continued operation against the risk of inconsistent or lost data. Kafka addresses this with replication, acknowledgments, idempotent writes, and transactions. section 5.0
Producer acknowledgment settings trade performance and tolerated loss against consistency:
- acks=0 sends without waiting for confirmation; it offers the highest performance when data loss is acceptable. section 5.1.1
- acks=1 waits for the leader to receive the message, providing an intermediate reliability level. section 5.1.1
- acks=all, also written acks=-1, waits until all in-sync replicas have replicated the message and gives the best data consistency. section 5.1.1
Choosing acknowledgments for a sensor stream
A temperature sensor reporting every millisecond may tolerate losing occasional readings when only the current value matters, making acks=0 attractive. For data that must be replicated consistently, configure a producer with --producer-property acks=all. section 5.1.1
Consistency also depends on compatible data definitions. Schema registries centralize schemas, support versioning, and enforce compatibility checks so producers and consumers can evolve without introducing breaking data changes. section 13.4
