A busy online store can outgrow a single log for every product update. Parallelism lets multiple workers process messages at the same time, but adding servers alone is not enough: the work must be divisible into independent pieces. Otherwise, spreading it across servers can initially make processing slower. section 4.2.1
Partitioning supplies those pieces. Product updates can go to separate partitions, while messages with the same key go to the same partition. This preserves the grouping needed for a product’s updates without requiring one order across every product. Likewise, a cab exchange can keep each city’s trip data together without requiring an order across cities. section 4.2.1
Try a two-partition product topic
Create a topic with `kafka-topics.sh --create --topic products.prices.changelog.multi-partitions --partitions 2 --replication-factor 1 --bootstrap-server localhost:9092`, then send product prices with `kafka-console-producer.sh --topic products.prices.changelog.multi-partitions --bootstrap-server localhost:9092`. A consumer reading from the beginning can display the six example updates in a different overall order from the producer input. For related updates, use producer keys instead: the keyed example sends repeated `coffee pads` updates with the same key. section 4.2.1
Kafka Streams extends this partition-based approach to processing. It assigns partitions of KStreams and KTables to tasks, its smallest processing units, then distributes tasks among application instances. Filtering a 12-partition topic creates 12 tasks: one instance handles all 12, while six instances typically handle two each. Tasks can also run on multiple threads within an instance, configured with `num.threads`. If an instance fails, tasks are redistributed to other instances. section 12.6
Figure 12.13 Kafka Streams applications are divided into tasks that are distributed across different threads on different instances. If an instance fails, another instance takes over the tasks.
The diagram traces how partition-based tasks spread across threads and instances, and how another instance takes over after a failure. section 12.6
Start by identifying which messages can be processed independently and which must stay together. Partition count determines the number of Kafka Streams tasks, so adding instances without adding partitions does not create more tasks. section 4.2.1 section 12.6
