Batch size specifies how many records or units are grouped for collective transmission or processing. In Kafka producers, batch.size sets the maximum batch size in client memory; the default is 16 KB. A full batch is sent when data continuously arrives, while linger.ms controls how long the producer waits for additional records when it is not yet full. section 3.2.5
Batch-size decisions balance performance against resource and delivery constraints:
- Larger batches can improve throughput by reducing overhead, but accumulation delays can increase latency. section 4.3.2
- Producer batches remain in client memory until acknowledged, so buffer.memory and batch.size must fit available resources. section 3.2.5 section 3.2.6
- For consumers, max.poll.records limits the number of messages fetched in one request, but variable message sizes make memory demand difficult to predict. section 4.3.2
- Workload patterns can change, requiring monitoring and periodic adjustment rather than a permanently optimal value. section 4.3.2
Continuous producer traffic
When records continuously arrive, a producer fills a batch up to batch.size and sends it. When traffic is sparse, linger.ms determines whether the producer waits for more records or sends the partially filled batch immediately. section 3.2.5
