A partition assignment maps Kafka topic partitions to the consumers responsible for processing them. Consumers sharing a group.id form one logical consumer, and Kafka assigns each topic partition to at most one group member. section 4.3.2
Use these rules to reason about assignments:
- A topic with six partitions and three consumers can assign two partitions to each consumer. section 4.3.2
- The number of partitions limits active consumers; additional consumers remain idle. section 4.3.2
- Different consumer groups receive independent assignments and do not affect one another. section 4.3.2
- Consumers periodically commit offsets so restarted members can resume processing from their previous position. section 4.7
Reassigning partitions after failure
If one of three consumers fails, Kafka rebalances its topic partitions across the two active consumers. When the failed consumer or a replacement rejoins, another rebalance can restore a more even distribution. section 4.3.2
Figure 4.9 Consumer groups allow the assignment of topic partitions across multiple consumers.
Inspect the consumer-group diagram to see how topic partitions are distributed across group members. section 4.3.2
Figure 4.10 The Kafka rebalance protocol reassigns topic partitions from failed consumers to currently active ones.
Inspect the rebalance diagram to follow partition movement from a failed consumer to active members. section 4.3.2
