A consumer service processing customer profiles and transactions may run several instances for capacity and resilience. If one instance fails, its partitions still need a processor; if another starts, the workload may need redistributing. Consumer group rebalancing is Kafka’s reassignment of partitions among group members when membership changes. Within a group, each partition is assigned to only one consumer at a time. section 4.0 section 4.2.7
A rebalance can follow several membership changes:
- A new consumer joins, potentially spreading partitions across more instances. section 4.2.7
- A consumer leaves gracefully, fails, or becomes too slow; its partitions are reassigned to healthy members. section 4.2.7
Figure 4.11 Redistribution of the partitions after a failure
The before-and-after partition assignment makes the failure case concrete: remaining consumers take over the failed instance’s partitions. section 4.2.7
Rebalancing preserves processing capacity after a failure, but it has a cost. An eager rebalance temporarily stops processing for all consumers, revokes their partitions, and assigns partitions again. The group coordinator manages membership and facilitates rebalances; a group leader manages partition assignments in coordination with it. section 4.2.7 section 4.7
Scaling a streaming application
Several Kafka Streams instances can share a group ID and process partitions from input topics. If an instance fails, its partitions move to active instances. Adding instances does not create unlimited parallelism: the number of input partitions limits how many instances can process them concurrently. section 8.1.2
Avoid treating every rebalance as useful. For temporary restarts, static membership gives each consumer a unique group.instance.id; if it rejoins within session.timeout.ms, it can retain its previous assignments. Long-running record processing may also require tuning max.poll.interval.ms and max.poll.records. Heartbeat and session timeouts balance prompt failure detection against unnecessary rebalances. Benchmark these settings against the application’s processing workload rather than choosing them solely for fast failover. section 4.2.8 section 4.3.3
