Consistency keeps a transaction within defined data rules, moving the system from one valid state to another without compromising integrity. section 7.4.2
In Kafka, consistency matters most in read-process-write pipelines. A transaction can read an Order event and atomically publish its related Shipment, Billing, and Inventory events. section 7.4.2
Order processing
An OrderService reads from ORDERS, processes product, payment, and shipping data, then writes to SHIPMENTS, BILLING, and INVENTORY. Exactly-once processing prevents retries or failures from duplicating or skipping downstream events. section 7.4.2
Consistency is an architectural choice with operational consequences:
- Favor data safety when financial transactions or payment processing cannot tolerate loss; accept downtime when necessary. section 11.3.2
- Favor availability for log collection or real-time analytics when remaining online matters more than preserving every message. section 11.3.2
- Improve durability with multiple in-sync replicas, acks=all, idempotence, manual offset commits, and read_committed consumers. section 11.3.2
Figure 7.24 OrderService reads, processes, and writes each event atomically.
Trace the atomic path from reading an order through processing it and producing all related events. section 7.4.2
Table 11.2 Balancing durability and availability
| Configuration setting |
Improve durability |
Improve availability |
Setting type |
|---|---|---|---|
|
|
Use |
With |
Producer |
|
|
Enable to avoid duplicate messages and improve reliability |
Enable to avoid duplicate messages and retries in the producer |
Producer |
|
|
Disable and manually commit offsets after processing to avoid data loss |
Consumer progresses without waiting for manual commit logic to complete |
Consumer |
|
|
Set to |
Set to |
Consumer |
|
|
Smaller batches reduce memory pressure and risk of data loss during failure |
Larger batches improve consumer throughput and avoid stalling (but may risk larger reprocesses) |
Consumer |
|
|
Reduce to catch slow consumers faster and avoid stale offsets |
Increase to allow slower consumers to stay in group longer |
Consumer |
|
|
Reduce to detect dead consumers quickly and maintain ISRs |
May be increased to avoid unnecessary rebalances, but for detecting dead consumers it can be decreased |
Consumer |
|
|
Use a higher replication factor for better data redundancy |
A lower replication factor can speed up recovery but risks data loss |
Broker |
|
|
Set higher to ensure more replicas are in sync before committing data |
Set lower to allow writes even with fewer in-sync replicas |
Broker |
|
|
Disable to avoid data loss with unclean leader election |
Enable to allow faster leader election even if some data is lost |
Broker |
|
|
Distribute replicas across racks to avoid rack failures causing data loss |
Ensure rack awareness for faster recovery if a rack fails |
Broker |
|
|
Reduce to flush data more frequently to disk, minimizing potential data loss |
Increase to flush data less frequently for faster throughput |
Broker |
|
|
Reduce to flush data to disk more frequently |
Increase to flush less often and improve broker throughput |
Broker |
|
|
Increase to reduce the window in which durability is weakened |
Increase to speed up broker recovery after a crash |
Broker |
|
|
Increase to reduce the risk |
Increase to maintain availability during maintenance or failures |
Broker |
|
|
Reduce to quickly detect lagging replicas and exclude them from ISRs |
Increase to tolerate more lag and avoid unnecessary ISR changes that can affect availability |
Broker |
|
|
Disable to keep leaders stable, avoiding leader changes that can affect |
Enable to allow automatic rebalancing of leaders, improving availability during broker failures |
Broker |
Compare producer and consumer settings that trade data durability against recovery speed and availability. section 11.3.2
