A transaction service may collect events throughout the day and load them into a data warehouse in one daily operation. This is batch processing: accumulating work over a period, then processing it as a finite group rather than handling each event as it arrives. It suits reporting, historical analysis, and other work that does not need an immediate result. section 5.2.9 section 5.1.4
Batch processing is an architectural choice, not simply a slower form of streaming. ETL commonly collects data in a staging area, transforms it, and periodically loads a target system. ELT also runs periodically, but transforms data after loading it into the target. Continuous stream processing instead handles incoming events as they arrive. For large volumes that do not require immediate analysis, scheduled processing may be simpler and more cost-effective. section 8.1.1 section 5.1.4
Figure 5.10 Time-based batch load to the data warehouse: the consumer buffers records and, at fixed intervals, bulk-loads a batch to the data warehouse (rather than per-message processing).
The time-based warehouse load shows a Kafka consumer buffering records and bulk-loading them at fixed intervals instead of loading each message separately. section 5.2.9
A Kafka-backed batch job needs explicit boundaries and failure handling. Records may be spread across partitions, so a job assembling a complete upload must gather them across those partitions. A timestamp can help locate the beginning of a period, but late-arriving events complicate its end. If only some uploaded records fail, the application needs its own way to identify and retry them: Kafka retrieves records sequentially, not as an arbitrary set of failed records. section 5.2.9
Kafka also batches messages for transport, even when the application processes events continuously. A producer groups serialized messages into per-partition batches in client memory and sends them in the background. The broker acknowledges a whole batch; an unacknowledged batch can be resent. On the other side, a consumer polls for groups of messages. Neither transport batch alone defines a daily job or a business processing window. section 3.2.5 section 4.1
Tune transport batching separately from the schedule of a batch job:
- For producers, batch.size limits batch size and linger.ms allows time for more messages to accumulate. Unsent batches reside in client memory and can be lost if the client crashes; treat messages as delivered only after acknowledgment. section 3.2.5
- For consumers, fetch.min.bytes and fetch.max.wait.ms trade response time against larger transfers. Fetch-size and poll-record limits help keep the returned volume manageable rather than overloading consumer memory. section 4.2.6
Choose the processing mode from the business requirement first. Nightly reconciliation, reporting, and batch scoring may naturally fit time-based windows. If a workflow does require continuous results, Kafka’s streaming model avoids imposing a batch boundary—but its client libraries may still batch network transfers for efficiency. section 10.1.3 section 2.4.3
