A consumer group lets Kafka treat multiple consumers as one logical consumer. Kafka assigns each topic partition to only one group member at a time, allowing consumption to scale across consumers. section 4.3.2
Use the group model to reason about assignments and failures:
- Set the same group.id on consumers that should share work; Kafka then places them in one group. section 4.3 section 4.3.2
- Divide partitions among active members. With six partitions and three consumers, each consumer receives two partitions. section 4.3.2
- Expect a rebalance when a consumer fails, misses the required poll timing, or joins again; Kafka redistributes the affected partitions. section 4.3.2
- Treat partition count as the scaling limit for active consumers in a group; additional consumers remain idle, while separate groups operate independently. section 4.3.2
A Java consumer using a group
Configure group.id, subscribe to the topic, and keep polling. The consumer processes each ProductTransaction returned by poll(); if processing succeeds, offset behavior follows the configured automatic or manual commit strategy. section 4.3
Figure 4.9 Consumer groups allow the assignment of topic partitions across multiple consumers.
View the partition-assignment diagram to see how consumer groups distribute topic partitions across members. section 4.3.2
Figure 4.10 The Kafka rebalance protocol reassigns topic partitions from failed consumers to currently active ones.
View the rebalance diagram to follow partition reassignment after a consumer fails. section 4.3.2
