Consumer group rebalancing redistributes topic-partition assignments when group membership changes. Consumers sharing a group ID act as one logical consumer: each partition is assigned to only one group member, while separate groups consume independently. The partition count caps active consumers; members beyond that count are idle. section 4.3.2 section 4.7
A rebalance commonly follows a consumer failure, a missed poll interval, or a new or returning member. If one of three consumers owning two partitions fails in a six-partition topic, its two partitions move to the active consumers. Kafka also detects inactive members through session timeouts and heartbeats, rather than always waiting for max.poll.interval.ms. section 4.3.2 section 4.3
Keep these operational distinctions in view:
- Set group.id to place consumers in the same logical group; assignment and rebalancing occur within that group. section 4.3
- Call KafkaConsumer.poll() within max.poll.interval.ms; exceeding it marks the consumer nonactive and triggers a rebalance. section 4.3
- Prefer the cooperative rebalance protocol in most cases because it stops only partitions being reassigned, reducing downtime compared with eager stop-the-world rebalancing. section 4.3.2 section 4.7
- Consider static membership where consumers often disconnect and return within a reasonable time, to provide greater group stability. section 4.7
A polling consumer and membership loss
A Java consumer subscribes to topic names and repeatedly calls poll(Duration.ofSeconds(5)) to process ProductTransaction records. Its group membership depends on continuing to poll within the configured maximum interval. If it stops polling long enough to become nonactive, Kafka redistributes its assigned partitions to other active consumers; after it returns, another rebalance can restore a balanced assignment. section 4.3 section 4.3.2
Rebalancing also carries application metadata in Kafka Streams. Instances with the same application ID can share their partition ownership and application.server information during rebalance. Interactive Queries uses that metadata to identify the instance responsible for a key’s partition and forward a query when the local instance does not own it; the query-serving and inter-instance communication layers still require implementation. section 13.2
Figure 4.14 Rebalancing with cooperative approach processing continues and only stops for partitions marked for reassignment.
Compare cooperative reassignment with eager rebalancing: unaffected partitions continue processing while only partitions marked for reassignment stop. section 4.3.2
