An online store cannot always wait for a scheduled report to discover a bad product price. Stream processing works on a continuous flow of events as they arrive, so a price change can be evaluated and acted on in near real time. section 12.1 section 12.1.2
Batch processing handles accumulated data at scheduled times and can produce a definitive result, but results may take hours. Stream processing handles events almost immediately. Its results must be treated as potentially intermediate: later events can change what the system knows. section 12.1.2
Responding to a price change
An event in a product-price changelog can prompt the store to correct or flag a suspicious price. A reaction to a price drop might update the website, notify customers, or alert the sales team. If the right response depends on a customer segment or pricing history, the processor also needs data from other topics. The aim is to act promptly without making unnecessary corrections. section 12.1 section 12.1.2
Figure 12.1 n batch processing, a large amount of data is processed at-once at specific times and the results are definite. When processing one event at a time, data is processed almost immediately, but it’s difficult to provide definite results, as the system is eventually consistent.
The diagram contrasts scheduled batch results with near-immediate, event-at-a-time processing whose results may change. section 12.1.2
Kafka can retain the incoming events while producers, consumers, and Kafka Connect move data between systems. Processing is a separate step: Kafka Streams or Apache Flink can evaluate data in Kafka and make processed results available to services. Persisted streams also let a processor resume after failure or work through historical events. section 12.0 section 17.5 section 1.1
Use stream processing when a decision benefits from reacting to changes quickly; keep batch processing where a scheduled, definitive report is more useful. In an established enterprise, plan for connections to existing systems rather than expecting a Kafka cluster alone to form a complete streaming platform. section 12.1.2 section 4.4
