A repartition topic is an intermediate Kafka topic that receives records after their keys change, allowing Kafka Streams to redistribute them to partitions before key-dependent processing. This matters when a stream is regrouped for aggregation or joined with another stream. section 7.2.3
Track repartitioning through the topology:
- selectKey changes the record key and marks the resulting stream as requiring repartitioning. section 7.2.3
- A later processor that uses the key, such as groupByKey or a join, triggers repartitioning; Kafka Streams adds sink and source nodes around the repartition topic. section 7.2.3
- Manual repartitioning clears the pending repartition flag, preventing later aggregations or joins from creating redundant repartition topics. section 7.2.3
- Kafka Streams can also remove redundant repartition topics through optimizations; repartitioning still adds network round trips and processing latency. section 7.2.3 section 7.2.5
- Increasing repartition partitions can increase task count only when keys are distributed relatively evenly. section 7.2.5
Explicitly creating a repartition topic
KStream<String, String> repartitioned = initialStream.repartition(Repartitioned.with(stringSerde, stringSerde).withName("multiple-aggregation").withNumberOfPartitions(10)); section 7.2.5
Figure 7.7 Redundant repartition nodes due to a key changing operation occurring previously in the topology
Inspect the topology to see how an earlier key-changing operation can produce redundant repartition nodes before aggregation and joining. section 7.2.3
