liveBook · Manning

Partition rebalancing

As explained in Kafka for Architects

Partition rebalancing is the migration and reassignment of Kafka partitions and their replicas across brokers to redistribute workload or remove capacity.

Partition rebalancing migrates and reassigns Kafka partitions and replicas across brokers to redistribute workload or remove capacity. Adding a broker does not automatically balance existing data; partitions must be moved, and leadership updated, across the cluster. section 9.2.1

Treat rebalancing as an operational change with measurable costs:

  • Moving partitions consumes source-broker CPU and disk I/O, uses network bandwidth, and can cause latency spikes. section 9.2.1
  • Removing a broker requires migrating its partitions first; abrupt loss leaves the cluster under-replicated and hot while replicas rebuild. section 9.2.1
  • Managed services may automate parts of rebalancing, while other providers require manual broker addition and data redistribution. section 9.2.1

Adding capacity

When a Kafka cluster gains a broker, capacity does not spread automatically. Operators migrate partitions to the new broker, update leadership, and account for transfer time, network use, and possible latency increases. section 9.2.1

How Kafka for Architects covers it

How deep it goes
Explains migration, leadership updates, performance costs, broker removal, and automation limits without giving an implementation walkthrough.
Examples
Uses broker addition, broker removal, managed-provider automation, replica rebuilding, and stale client metadata as operational examples.
What you can do after
Operate/configure Kafka scaling with awareness of migration costs, replication risk, latency effects, and provider automation limits.
How it connects
Connects Kafka brokers, partition leadership, replicas, cloud scaling, managed Kafka, and Cruise Control.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.