When several application instances consume the same Kafka topic, they need a way to share partitions and respond when an instance joins or fails. Each consumer group has a group coordinator: a Kafka broker that manages group membership and coordinates partition assignments. Different groups may have different coordinators; if one becomes unreachable, a new coordinator is elected. section 4.2.3 section 4.7
Consumers first discover their coordinator, then ask it to join the group. In the group-leader assignment flow, one consumer becomes the group leader. It receives member and subscription information, calculates assignments using the configured partition assignment strategy, and returns them to the coordinator. The coordinator distributes the assignments. Consumers need not contact one another; after assignment, they fetch records directly from partition leaders. section 4.2.3
Figure 4.9 Distributing the assignments
Shows the coordinator distributing partition assignments after the group leader calculates them. section 4.2.3
Membership changes can require a rebalance. A joining consumer may take over some partitions; a departing or unhealthy consumer leaves partitions for healthy members to process. An eager rebalance pauses all consumers while assignments are rebuilt, whereas a cooperative rebalance changes ownership incrementally. The coordinator can detect an unhealthy member and remove it from the group. section 4.2.7
Under the next-generation consumer rebalance protocol, assignment calculation moves to the broker. Consumers send the coordinator membership, subscription, currently owned partition, timeout, and preferred assignor information through the ConsumerHeartBeat API. The coordinator returns a target assignment and heartbeat interval when needed; in a stable group, the request serves mainly as a heartbeat. Server-side assignors and heartbeat limits are broker-side configuration choices. section 4.2.10
A worker disappears
Kafka Connect workers sharing a group.id send periodic heartbeats to the group coordinator. If a worker stops heartbeating, the coordinator detects its failure and its tasks are redistributed to active workers. This is worker failure handling, not automatic recovery from every task failure: a task that enters FAILED state must be monitored and manually restarted after its cause is resolved. section 7.3.2
