When one application cannot process incoming events quickly enough, Kafka can divide the work by partitioning a topic. Consumers then process different partitions concurrently. The partition count sets the maximum number of consumer instances that can read that topic in parallel within one consumer group. section 2.2.1
A producer sends each message to a partition, where it is appended with an offset. This supports ordering within a partition, not across the whole topic. Partition count is therefore both a capacity decision and an ordering constraint. section 2.2.1
Consumers in the same group share the work: each partition belongs to only one group member at a time. Separate groups can independently read the same topic. For example, several instances of an address-change service can collaborate as one group, while another service uses its own group. section 4.1 section 4.2.1
Figure 4.4 Assignment of the topic partitions to the consumer instances within a consumer group
The partition-assignment diagram shows how members of one consumer group divide a topic’s work. section 4.2.1
More consumer instances cannot overcome a partition-count ceiling. Consumer lag helps reveal a group’s processing backlog when deciding whether to scale. Another option is to dispatch records to a bounded worker pool sharded by key: different keys can run concurrently while each key stays ordered. Because workers may finish out of order, commits can advance only through the highest contiguous completed offset. section 4.3.1 section 4.3.1
Kafka Streams applies the same partition-based limit to distributed processing. Instances sharing a consumer group run independent tasks that combine input-partition assignments, a processing topology, and any state stores. Tasks can run on different threads or instances; if an instance fails, its tasks are reassigned. For multiple input topics, the topic with the most partitions determines the application’s parallelism. section 8.1.2 section 8.3.4
