Repartition
Repartitioning is the redistribution of records or items across partitions according to a partitioning rule, potentially changing the number of partitions.
Kafka Streams in Action, Second Edition
KStream.repartition is a Kafka Streams operation that explicitly redistributes stream records into a repartition topic using configurable partitioning settings.
- How it is used
- Use KStream.repartition to explicitly redistribute changed keys before key-dependent aggregations or joins.
- How deep it goes
- Explains topology nodes, sub-topologies, configuration options, scaling effects, and optimization behavior through Java examples.
- Examples
- Demonstrates selectKey followed by repartition, groupByKey counting, joining, and a ten-partition throughput configuration.
- What you can do after
- Apply it via KStream.repartition to control repartition topics, downstream stateful processing, and task parallelism.
Read how Kafka Streams in Action, Second Edition explains repartition →
Apache Kafka in Action
Repartitioning redistributes stream records according to their keys so that records requiring keyed processing reach the appropriate task.
- How it is used
- Ensures keyed joins and aggregations run on the task corresponding to each record’s key after a key-changing operation.
- How deep it goes
- Explains the internal topic and two-subtopology data flow, including resource costs, but gives no tuning procedure.
- Examples
- Traces selectKey() followed by count() and contrasts map() with mapValues() using product-price records.
- What you can do after
- Apply it via Kafka Streams by choosing mapValues() when the key should stay unchanged and anticipating repartition costs before keyed operations.
Fast Python: High performance techniques for large datasets
- How it is used
- Uses repartitioning to tune Dask execution granularity as workload intensity changes across a computation.
- How deep it goes
- Explains partition-count and index-division strategies, task-graph effects, metadata limits, and data-movement costs.
- Examples
- Demonstrates tax data, three-to-two partition reduction, task-graph visualization, and Geo_Name-based index divisions.
- What you can do after
- Apply repartitioning via Dask to change partition counts or index ranges while evaluating balance, metadata, memory, and network costs.
Read how Fast Python: High performance techniques for large datasets explains repartition →
Related concepts
- repartition topicproduces · A repartition topic is an intermediate Kafka topic that redistributes records by key before downstream keyed stream processing.
- optimizationaddressed by · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
- dataset splituses · A dataset split is the division of a dataset into distinct subsets assigned different purposes, such as training, validation, or testing.
- key-changing operationused by · A key-changing operation is a stream-processing operation that assigns or derives a different key for records.
- Daskuses · Dask is an open-source Python library for parallel and distributed computing using task graphs and scalable data structures for large or partitioned workloads.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.