Uptime describes keeping a system operational. In Kafka, a partition can go offline when too few in-sync replicas remain; this prevents producers from writing but avoids risking data loss. section 11.3.2
The acceptable trade-off depends on the application:
- Prioritize durability for financial transactions and payment processing, where downtime is preferable to risking data loss. section 11.3.2
- Prioritize availability for log collection and real-time analytics dashboards, where staying online may matter more than avoiding small losses. section 11.3.2
Table 11.2 Balancing durability and availability
| Configuration setting |
Improve durability |
Improve availability |
Setting type |
|---|---|---|---|
|
|
Use |
With |
Producer |
|
|
Enable to avoid duplicate messages and improve reliability |
Enable to avoid duplicate messages and retries in the producer |
Producer |
|
|
Disable and manually commit offsets after processing to avoid data loss |
Consumer progresses without waiting for manual commit logic to complete |
Consumer |
|
|
Set to |
Set to |
Consumer |
|
|
Smaller batches reduce memory pressure and risk of data loss during failure |
Larger batches improve consumer throughput and avoid stalling (but may risk larger reprocesses) |
Consumer |
|
|
Reduce to catch slow consumers faster and avoid stale offsets |
Increase to allow slower consumers to stay in group longer |
Consumer |
|
|
Reduce to detect dead consumers quickly and maintain ISRs |
May be increased to avoid unnecessary rebalances, but for detecting dead consumers it can be decreased |
Consumer |
|
|
Use a higher replication factor for better data redundancy |
A lower replication factor can speed up recovery but risks data loss |
Broker |
|
|
Set higher to ensure more replicas are in sync before committing data |
Set lower to allow writes even with fewer in-sync replicas |
Broker |
|
|
Disable to avoid data loss with unclean leader election |
Enable to allow faster leader election even if some data is lost |
Broker |
|
|
Distribute replicas across racks to avoid rack failures causing data loss |
Ensure rack awareness for faster recovery if a rack fails |
Broker |
|
|
Reduce to flush data more frequently to disk, minimizing potential data loss |
Increase to flush data less frequently for faster throughput |
Broker |
|
|
Reduce to flush data to disk more frequently |
Increase to flush less often and improve broker throughput |
Broker |
|
|
Increase to reduce the window in which durability is weakened |
Increase to speed up broker recovery after a crash |
Broker |
|
|
Increase to reduce the risk |
Increase to maintain availability during maintenance or failures |
Broker |
|
|
Reduce to quickly detect lagging replicas and exclude them from ISRs |
Increase to tolerate more lag and avoid unnecessary ISR changes that can affect availability |
Broker |
|
|
Disable to keep leaders stable, avoiding leader changes that can affect |
Enable to allow automatic rebalancing of leaders, improving availability during broker failures |
Broker |
Compare Kafka producer and consumer settings for favoring durability or faster recovery, and consider the resulting risk of downtime or message loss. section 11.3.2
