Suppose an online store publishes successive prices for each product. If updates for one product land in different Kafka partitions, a consumer may not see them in the order they were produced. Give each message a product-name key to keep that product’s updates together. The key is an optional part of the message; the value carries the price. A topic needs no special setting to accept keyed messages. section 3.2.3 section 4.2.1
Send and inspect keyed prices
For an existing topic, run `kafka-console-producer.sh --topic products.prices.changelog.keys --bootstrap-server localhost:9092 --property parse.key=true --property key.separator=:`. Enter `coffee pads:10`, `cola:2`, then `coffee pads:11`. The colon separates each key from its value. To see both fields, run `kafka-console-consumer.sh --from-beginning --topic products.prices.changelog.keys --bootstrap-server localhost:9092 --property print.key=true --property key.separator=:`. Without `print.key=true`, the console consumer displays only the prices. section 3.2.3
A producer’s partitioner hashes the key and takes the result modulo the topic’s partition count. The same key therefore selects the same partition under a consistent partitioning scheme. Kafka guarantees order within a partition, not across partitions; the stated ordering guarantee for same-key messages is for a single producer. If order between messages does not matter, a key can be omitted. section 4.2.1 section 3.3
Choose and maintain keys with these consequences in mind:
- Increasing a topic’s partition count changes the modulo calculation. Later messages for a key may move to another partition while older ones remain, disrupting their order until the old data expires. If that is unacceptable, plan a new topic and migration instead. section 6.1.3
- Mixed Java and librdkafka producers can hash the same key differently. Set the librdkafka partitioner to `murmur2_random` to align their partitioning. section 4.2.1 section 8.1.1
- Keys also enable log compaction: a compacted topic requires them and retains at least the latest entry for each key rather than keeping every outdated value. section 10.4
from confluent_kafka import Producer
producer = Producer({
'bootstrap.servers': 'localhost:9092', #1
'acks': -1, #1
'enable.idempotence': True, #1
'partitioner': 'murmur2_random', #2
})
def delivery_report(err, msg): #3
if err is not None: #4
print(f"Delivery failed for record {msg.key()}: {err}")
return
print(f"Record {msg.key()} successfully produced to {msg.topic()}
[{msg.partition()}] at offset {msg.offset()}")
producer.produce(
"products.prices.changelog", #5
key="cola",
value="2",
on_delivery=delivery_report) #6
The Python producer example passes `key="cola"` with `value="2"` and configures `murmur2_random` for compatible partitioning. section 8.1.1
