liveBook · Manning

Round-robin load balancing

As explained in Apache Kafka in Action

Round-robin distribution assigns successive records or work items to available partitions or recipients in cyclic order.

Suppose an online store sends price updates to a Kafka topic with multiple partitions. When a producer omits the message key, it can distribute messages across partitions in round-robin fashion rather than keeping updates for one product together. This helps spread producer traffic; if partitions are distributed evenly across brokers, broker load can be reasonably even too. section 4.2.1 section 4.2.2

Think of the basic pattern as partition 0, then partition 1, and so on. That does not mean every message alternates partitions in current Kafka producers: since Kafka 2.4, the optimized procedure selects another partition after sending a batch. Filling batches better reduces network requests and can improve throughput and compression. section 4.2.1

Price updates without keys

Create a two-partition topic with `kafka-topics.sh --create --topic products.prices.changelog.multi-partitions --partitions 2 --replication-factor 1 --bootstrap-server localhost:9092`. Then run `kafka-console-producer.sh --topic products.prices.changelog.multi-partitions --bootstrap-server localhost:9092` and enter successive updates such as `coffee pads 10`, `cola 2`, and `coffee pads 11`. These lines have no product keys, so updates for coffee pads need not share a partition. section 4.2.1

$ kafka-console-consumer.sh \
    --topic products.prices.changelog.multi-partitions \
    --from-beginning \
    --bootstrap-server localhost:9092
cola 2
coffee pads 11
coffee pads 10
energy drink 3
coffee pads 10
coffee pads 12
# Press Ctrl-C to cancel
Processed a total of 6 messages

The consumer output differs from the order in which the price updates were entered. Distributing records across partitions does not preserve one overall reading order. section 4.2.1

Use keys when order matters for a particular product: messages with the same key go to the same partition, preserving their sequence there. Round-robin distribution addresses placement, not downstream processing capacity. A single consumer reading every partition can still become a bottleneck; consumer groups let instances of one service split that work. section 3.2.3 section 4.2.2

How Apache Kafka in Action covers it

How deep it goes
Explains cyclic placement and Kafka 2.4's batch-level optimization, but does not specify a detailed partitioner algorithm.
Examples
Uses a two-partition price-update topic, console-producer commands, and consumer output that differs from production order.
What it assumes
Introduces keyless placement through topic partitions and price updates; assumes familiarity with running Kafka console commands.
What you can do after
Operate/configure it by omitting producer keys when per-product ordering is unnecessary, and choose keys when that ordering matters.
How it connects
Connects message keys and partition ordering to batching, brokers, and consumer groups.

Also explained in

Compare how 3 books explain round-robin load balancing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.