A Kafka consumer sends periodic heartbeats to its group coordinator to signal that it is active. If a consumer fails unexpectedly, missed heartbeats give the coordinator a way to detect the failure and rebalance its partitions among the remaining consumers. section 4.2.7
Three settings distinguish liveness from progress:
- heartbeat.interval.ms sets how often the consumer sends a heartbeat. section 4.2.7
- session.timeout.ms sets how long the coordinator waits without a heartbeat before treating the consumer as failed and initiating a rebalance. It must exceed the heartbeat interval; a typical setting is three times as long. section 4.2.7
- max.poll.interval.ms limits the time between consumer polls. If processing takes too long, the consumer leaves the group and triggers a rebalance itself—even though this is a different condition from missed heartbeats. section 4.2.7
A failed worker versus a failed task
Kafka Connect workers also heartbeat to a group coordinator. When a worker fails, its missing heartbeats lead to task redistribution across active workers. A task that enters FAILED because of a database constraint error is different: it does not restart automatically and needs monitoring, a fix, and a manual restart. section 7.3.2
Heartbeat also describes liveness signals in cluster management, but the participants and consequences differ. Brokers signal liveness to the leader controller; controller fetch requests implicitly confirm leader availability. In ZooKeeper-managed clusters, brokers heartbeat to ZooKeeper, which can mark a broker unavailable and prompt partition-leader reassignment. Keep these mechanisms distinct from consumer-group heartbeats when diagnosing a rebalance or failover. section 9.1 section 9.1.3
