A producer sending 10 KB records may get little benefit from Kafka’s default 16 KB batches. Sending more records together can reduce network requests, but the right size depends on the message rate and the latency the application can tolerate. section 6.2.1 section 6.2.2
The producer setting `batch.size` limits a batch’s size in bytes; its default is 16,384 bytes. The producer buffers records by topic and partition, forms batches, and sends them to the appropriate brokers. `linger.ms` sets how long it may wait for more records. Even when that wait is zero, records can be batched if they arrive faster than individual sends complete. section 6.2.1 section 8.1.2
Figure 6.5 The producer batches messages for each topic and partition. A message is sent as soon as either the batch is full or the linger time is up.
The diagram shows the two triggers for sending a partition’s batch: it reaches the size limit, or its linger time expires. section 6.2.1
Measure before and after tuning
A producer performance test with one million 10 KB records initially reported about 117 MB/s and 165 ms average latency. With `batch.size=100000` and `linger.ms=100`, it reported 437.55 MB/s and 64.61 ms. These are results from changing both settings together, not a guaranteed gain from batch size alone. section 6.2.2
$ kafka-producer-perf-test.sh \
--topic performance-test \
--num-records 1000000 \
--record-size 10000 \
--throughput -1 \
--producer-props bootstrap.servers=localhost:9092 \
batch.size=100000 linger.ms=100
...
1000000 records sent, 45879.977978 records/sec (437.55 MB/sec),
64.61 ms avg latency, 279.00 ms max latency, 58 ms 50th,
101 ms 95th, 126 ms 99th, 178 ms 99.9th.
The runnable test passes both settings through `--producer-props` and reports throughput and latency for the tuned run. section 6.2.2
Larger batches generally improve throughput, while waiting to fill them can add latency. A too-small batch can also hurt latency when records pile up, as in the test above. Compare repeated tests using production-like records and conditions rather than adopting one benchmark’s values unchanged. section 6.2.1 section 6.2.2
