Sending every Kafka record separately creates network overhead. Batching groups records so a producer can send fewer, larger requests. It improves throughput, especially when many small records are produced, but waiting to fill a batch can add latency. section 6.2.1 section 8.3.3
After serialization and partition selection, the producer places records in a buffer. It creates batches for each topic and partition, then sends partition-based batches to the relevant leader broker. A batch is sent when it fills or its waiting time expires. With the default wait of 0 ms, batching can still happen when records arrive faster than they can be sent. section 8.1.2 section 6.2.1
Two producer settings control the main trade-off:
- batch.size limits batch size in bytes; its stated default is 16 KB. A larger limit can reduce requests when enough records arrive. section 6.2.1
- linger.ms limits how long the producer waits for more records; its stated default is 0 ms. Increasing it can improve throughput when immediate delivery matters less, but may increase latency. section 6.2.1 section 6.5
Test with production-like records
Run kafka-producer-perf-test.sh against a test topic, then compare throughput and latency after adding --producer-props batch.size=100000 linger.ms=100. In one test with 10 KB records, throughput rose from about 117 MB/s to 437.55 MB/s, while average latency fell from about 165 ms to 65 ms. The default 16 KB batch was too small to batch those records effectively; the result is not a promise that waiting longer always lowers latency. Test with your own data and cluster conditions. section 6.2.2
A batch also persists as a unit in a broker log segment. kafka-dump-log.sh can show its first and last offsets, record count, size, position, timestamp, and checksum. Compression, when enabled, operates on whole batches: brokers store the compressed batches, and consumers unpack them. This saves bandwidth but can affect latency. section 8.3.3 section 6.2.1 section 6.5
Figure 6.5 The producer batches messages for each topic and partition. A message is sent as soon as either the batch is full or the linger time is up.
The diagram shows separate topic-and-partition batches and the two triggers for sending one: size or elapsed linger time. section 6.2.1
Producer batching is not the same as scheduled batch processing. A nightly job may process a day of transactions at once; producer batching groups records for transmission while they flow into Kafka. section 12.1.2 section 6.2.1
