Consumer group

As explained in Kafka for Architects

A shared consumer group is a Kafka consumption mode that permits multiple consumers in a group to receive records from the same partition.

A consumer group lets multiple consumers collaborate on reading Kafka topics. In the classic consumption model, each partition is assigned to only one member of a group at a time. Members divide the work; a different group can read the same topic independently. A service often uses one group, but a group need not map to exactly one service. section 4.1 section 4.2.1

Partition count limits how far a classic group can scale: adding consumers cannot create more simultaneous partition assignments than there are partitions. One consumer can also handle every partition when parallel processing is unnecessary. Where records sharing a key must remain ordered, routing them to one partition keeps their processing with one group member; changing the partition count can disrupt that routing. section 2.2.2 section 2.2.3 section 4.3.1

Figure 4.4 Assignment of the topic partitions to the consumer instances within a consumer group

The partition-assignment diagram shows how group members divide a topic without assigning a partition to two members in the classic model. section 4.2.1

A customer-data consumer service

Consumers reading customer profiles and transactions can share a group.id while subscribing to different topics. Distinct client.id values help monitor and debug them. Group coordination does not, however, make concurrent updates to the same database aggregate safe: the two sets of consumers can overwrite one another, requiring a separate safeguard such as optimistic locking. section 4.2.2 section 4.2.13

Committed offsets are tracked per group, so committing a record's position needs care: marking it read does not necessarily mean processing finished. Consumer lag indicates how far a group is behind and helps guide scaling decisions. Failures or membership changes can trigger a rebalance that redistributes partitions. section 4.1 section 4.2.4 section 4.3.1 section 8.1.2

Record the intended consumption model when designing a topic. The one-member-per-partition rule describes classic groups; shared consumer groups, introduced in Kafka 4.0, permit a partition to be assigned to multiple consumers within a group. That distinction changes the assignment constraint and should not be inferred from group.id alone. section 10.1.3

How Kafka for Architects covers it

How deep it goes

Explains classic partition assignment, group offsets, lag, and rebalances; identifies shared groups without detailing their mechanics.

Examples

Uses customer-data consumers, balance updates, and partition-assignment diagrams to illustrate collaboration and ordering.

What it assumes

Assumes familiarity with topics, partitions, and consumer clients while defining how consumers collaborate in a group.

What you can do after

Operate/configure Kafka consumers with group.id and use partition count and lag to assess scaling.

How it connects

Connects topic partitions and publish-subscribe consumption to consumer groups, with offsets and consumer lag shaping operations.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Apache Kafka in ActionA consumer group is a set of consumers that collaboratively consume topic partitions, with each partition assigned to one consumer in the group.
• 

  liveBookmanning.com
  Kafka Streams in Action, Second EditionA consumer group is a set of Kafka consumers that jointly consume a topic's partitions, with each partition assigned to at most one group member at a time.
• 

  liveBookmanning.com
  Troubleshooting Java, Second EditionA consumer group is a set of consumers that cooperatively processes the same Kafka topic stream.

Compare how 4 books explain consumer group →

Related concepts

• group.idconfigured by · group.id is a Kafka consumer configuration property that identifies the consumer group whose partition assignments and consumption progress the consumer participates in.
• offsetuses · An offset is a position or displacement measured relative to a reference point, such as within an ordered sequence, memory region, or layout.
• partitionuses · A partition is a distinct subdivision of a larger data, storage, or work unit that can be stored, managed, or processed separately.
• Apache Kafkaused by · Apache Kafka is a distributed event-streaming platform for publishing, storing, processing, and consuming streams of records.
• publish/subscribe modelimplements · The publish-subscribe model is a messaging pattern in which publishers send messages without addressing specific recipients, and subscribers receive messages according to their interests or subscriptions.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.