A customer-data service may need to process profile and transaction events concurrently while keeping related updates in order. A Kafka topic groups the events; partitions divide that topic into independently processed units. Choosing the partition count is therefore both a throughput decision and an ordering decision. section 2.1 section 2.2.1
Each partition has an integer index, starting at 0, and is identified by its topic name and index. A producer directs a record to a partition, where it is appended to an immutable sequence. The record’s offset identifies its position within that partition. Order is maintained within a partition, not across the whole topic. section 2.2.1
Figure 2.5 Utilizing partitions for parallel consumption. In this setup, each thread within a service instance is allocated its own partition, with the total number of partitions determining the maximum degree of parallelization achievable.
The diagram shows how separate partition assignments allow consumers to process one topic in parallel. section 2.2.2
Partition design has three practical consequences:
- The partition count caps the number of consumers that can process a topic in parallel within one consumer group. One consumer can handle multiple partitions; a partition belongs to only one consumer in that group at a time. section 2.2.2 section 4.1
- A message key can direct related records to the same partition, preserving their relative order. Records in different partitions have no guaranteed order relative to one another. section 2.7
- Partitions are replicated across brokers for redundancy. If too few replicas remain in sync during a failure, a partition can go offline; durability and availability settings must reflect the application’s tolerance for downtime or data loss. section 2.7 section 11.3.2
Customer-data processing
Suppose a service aggregates customer updates for an operational dashboard. Several consumers in one group can share its topic’s partitions, each processing an assigned portion. If processing falls behind, consumer lag indicates the backlog for a partition; adding consumers helps only up to the parallelism allowed by the partition count. section 2.1 section 4.2.1 section 4.3.1
Plan partition counts against expected consumer workload rather than assuming that more consumers will always increase throughput. Managed Kafka may still require partition configuration; serverless Kafka can instead handle partitions and capacity planning behind the scenes. section 4.3.1 section 12.2.2
