liveBook · Manning

Repartition

As explained in Apache Kafka in Action

Repartitioning redistributes stream records according to their keys so that records requiring keyed processing reach the appropriate task.

Suppose product-price records arrive without keys. A map() operation can make each product ID the key. Keys keep related records in a partition, but changing a key does not by itself move a record to the partition for its new key. section 12.2.1 section 12.6

Kafka Streams marks a stream for repartitioning after a key-changing operation such as map() or selectKey(). Stateless operations such as filter() and mapValues() can continue without moving records. Before a later join or aggregation, however, Kafka Streams writes records to an internal topic ending in -repartition and reads them back so each key reaches the appropriate task. section 12.6

Change the key, then count

A selectKey() followed by count() requires repartitioning before the count. The processing topology becomes two subtopologies: one writes to the repartition topic, and the other reads from it and performs the aggregation. section 12.6

Figure 12.14 If we first execute a selectKey() and then an aggregation such as count(), the topology needs to be split up into multiple subtopologies.
A close-up of several boxes

AI-generated content may be incorrect.

The diagram shows where a selectKey()-then-count() pipeline splits to move records by their new keys. section 12.6

This routing makes keyed joins and aggregations correct, but the extra write and read sharply increase network traffic and storage needs. If only a value needs changing, prefer mapValues() over map(): preserving the key can avoid that cost. section 12.6 section 12.2.1

How Apache Kafka in Action covers it

How deep it goes
Explains the internal topic and two-subtopology data flow, including resource costs, but gives no tuning procedure.
Examples
Traces selectKey() followed by count() and contrasts map() with mapValues() using product-price records.
What it assumes
Builds on previously introduced selectKey(), map(), partitions, and tasks rather than defining those operations from scratch.
What you can do after
Apply it via Kafka Streams by choosing mapValues() when the key should stay unchanged and anticipating repartition costs before keyed operations.
How it connects
Links key-changing processors to partitions, tasks, joins, aggregations, internal topics, and subtopologies.

Also explained in

Compare how 3 books explain repartition →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.