A single server can become a bottleneck when a service grows. A dataset split divides data among subsystems so they can handle more work in parallel. In a worldwide cab service, trips from the same city may need to stay together, while trips from different cities can be stored and processed separately. This is partitioning, also called sharding in database systems. Splitting data helps only when the work itself can be parallelized. section 4.2.1
Kafka applies this idea to topics. For an online store’s price changes, each product’s messages should retain their order, but changes to different products need no shared order. A producer can use the product as the message key to send messages with that key to the same partition. Kafka selects the partition using the key’s hash and the number of partitions; consumers reading across partitions should not expect one global message order. section 4.2.1 section 6.1.3
# First create the topic products.prices.changelog.multi-partitions-keys as
above
$ kafka-console-producer.sh \
--topic products.prices.changelog.multi-partitions-keys \
--property parse.key=true \
--property key.separator=":" \
--bootstrap-server localhost:9092
> coffee pads:10
> cola:2
> cola:1
> energy drink:3
> coffee pads:11
> coffee pads:12
> energy drink:4
> coffee pads:10
# Press Ctrl-D to cancel
After creating the topic, this console-producer example sets `parse.key=true` and uses `:` to separate each product key from its price. Repeated updates to a product therefore carry the same key. section 4.2.1
Choose the split with its operating consequences in mind:
- More partitions can distribute load and let consumers process messages in parallel, but partition counts also affect client RAM use, broker file descriptors and operational complexity. section 6.1 section 6.5
- Increasing a topic’s partition count changes the key-to-partition calculation. New messages for a key may land apart from older ones, disrupting their order until the old data expires. section 6.1.3
- If that ordering risk is unacceptable, migrate to a new topic with the desired partition count rather than changing the existing topic in place. Producers using different client libraries must also use compatible partitioning algorithms. section 6.1.3 section 14.0
