liveBook · Manning

Repartition

As explained in Apache Kafka in Action

Repartitioning redistributes stream records according to their keys so that records requiring keyed processing reach the appropriate task.

Repartitioning redistributes stream records by key so that a join or aggregation processes related records in the appropriate Kafka Streams task. It matters when a processing step changes keys: the records’ existing partition placement may no longer match their new keys. section 12.6

Operations such as selectKey() and map() can change keys and mark a stream or table for repartitioning. Stateless operations such as filter() and mapValues() do not require redistribution just because an earlier operation changed the key. A subsequent join or aggregation does require correct key-based placement. section 12.6

Kafka Streams writes the data to an internal topic whose name ends in -repartition, then reads it back before the keyed operation. This divides the topology into subtopologies: one writes to the internal topic, and the next reads from it and continues processing. The extra write and read increase network traffic and storage requirements. section 12.6

Changing a key before counting

A topology that calls selectKey() and then count() needs repartitioning between the key change and the aggregation. The first subtopology selects the key and writes to the internal topic; the second reads the redistributed records and performs the count. section 12.6

Figure 12.14 If we first execute a selectKey() and then an aggregation such as count(), the topology needs to be split up into multiple subtopologies.
A close-up of several boxes

AI-generated content may be incorrect.

Follow the split between key selection and aggregation through the repartition topic. section 12.6

Avoid changing the key when only the value needs updating. For example, mapValues() can convert a price from dollars to cents without changing its key; using map() to assign a new key may lead to a costly repartition later. section 12.2.1

How Apache Kafka in Action covers it

How deep it goes
Explains automatic repartition topics, topology splitting, and network and storage costs, without detailing an explicit repartition() call.
Examples
Uses selectKey() followed by count(), and contrasts price conversion with assigning a product key.
What you can do after
Evaluate when a key change requires redistribution and when a value-only operation can avoid it.
How it connects
Connects key changes with partitions, joins, aggregations, tasks, and subtopologies.

Also explained in

Compare how 3 books explain repartition →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.