Consumer group rebalancing

As explained in Apache Kafka in Action

A rebalance is the redistribution of consumer-group partition assignments after membership, partition, or consumer availability changes.

A consumer group divides a topic’s partitions among its consumers. When a consumer joins or fails, or the topic’s partitions change, the group needs a new distribution. That redistribution is a rebalance. Kafka brokers coordinate the group, while consumers take part in deciding who receives each partition. section 9.4.1 section 9.5

Each group has a broker acting as its group coordinator. Consumers send join requests to that coordinator; during a rebalance, one consumer acts as group leader, determines the partition assignments, and communicates them to the coordinator. The other consumers follow those assignments. Kafka also uses the rebalance protocol to distribute resources such as Kafka Streams and Kafka Connect tasks. section 9.4.1 section 9.4.2

Choosing an assignment strategy

Suppose three consumers subscribe to two topics with two partitions each. With the default Range Assignor, one consumer gets partition 0 of both topics, another gets partition 1 of both, and the third gets none. Keeping matching partitions together can help when data from the two topics must be joined, but it leaves a consumer idle. When cross-topic joins are not needed, Round Robin Assignor spreads partitions more evenly. Configure the choice with partition.assignment.strategy, and use the same strategy throughout the group; mismatched strategies can cause unpredictable assignment changes. section 9.4.2 section 9.5

Figure 9.5 By default, partitions in a consumer group are assigned in such a way that the same partition across different topics is always consumed by the same consumer. While this may not be the most resource-efficient allocation, it allows us to easily perform joins across multiple topics.

Compare the matching cross-topic partitions with the idle consumer in the Range Assignor example. section 9.4.2

A rebalance can interrupt consumption, depending on the assignment strategy. Cooperative Sticky Assignor reduces that disruption by moving partitions in two rounds: consumers first release partitions that must move, then the unassigned partitions are allocated. Other consumers can continue consuming during the process. Static membership offers another way to reduce rebalance frequency, using unique group instance IDs and longer session timeouts. section 9.4.2 section 9.5

Figure 9.7 The Cooperative Sticky Assignor avoids stopping the world and rebalances by replacing a single rebalance by two consecutive ones. During the first rebalance, consumers are asked to let go of the partitions that should be moved to another consumer; in the second rebalance, the now dangling partitions are assigned to the consumers.

Trace the release and reassignment rounds that avoid stopping the whole group. section 9.4.2

Watch for rebalances caused by slow consumers. If processing delays polls beyond max.poll.interval.ms, a consumer can be considered inactive and trigger a group-wide rebalance. Monitor time between polls and time since the last poll alongside rebalance counts, failures, and latency. Some rebalancing is normal; rising frequency or duration deserves investigation, especially when consumption pauses during reassignment. section 15.3.3

How Apache Kafka in Action covers it

How deep it goes

Explains broker and consumer roles, assignment strategies, disruption-reduction methods, and monitoring; does not specify a complete protocol message sequence.

Examples

Compares three consumers sharing two two-partition topics under Range Assignor and illustrates Cooperative Sticky Assignor’s two-round reassignment.

What it assumes

Builds on consumers, topics, partitions, and consumer groups while introducing their coordination roles.

What you can do after

Operate/configure it by choosing a consistent assignment strategy and monitoring polling delays, rebalance failures, and latency.

How it connects

Connects consumer groups and partition assignment to stream joins, static membership, consumer polling, and monitoring metrics.

liveBookmanning.com

Read Apache Kafka in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Kafka Streams in Action, Second EditionA consumer group rebalance is the redistribution of Kafka topic partitions among consumers in a consumer group.
• 

  liveBookmanning.com
  Kafka for ArchitectsConsumer group rebalancing is the redistribution of topic-partition assignments among consumer group members when membership or partition availability changes.

Compare how 3 books explain consumer group rebalancing →

Related concepts

• rolling updateproduced by · A rolling update is a deployment or upgrade technique that updates or replaces instances or resources incrementally rather than all at once, allowing the rest of the system to continue operating during the change.
• static group membershipaddressed by · Static group membership is a consumer-group mode in which instances retain stable identities across restarts to avoid unnecessary rebalancing.
• group coordinatorproduced by · A group coordinator is the Kafka broker responsible for managing a consumer group’s membership, liveness, partition assignments, and committed offsets.
• Kafka Streamsused by · Kafka Streams is a client library for building applications that continuously process data from and write data to Apache Kafka topics.
• partitionproduced by · A partition is a distinct subdivision of a larger data, storage, or work unit that can be stored, managed, or processed separately.

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.