An order can trigger inventory, payment, shipping, and analytics work as soon as it is placed. Stream processing goes further than delivering that event: it continuously processes an unbounded flow of events as they arrive and emits results with minimal latency. section 1.0 section 8.1
The contrast is with periodic batch pipelines. ETL transforms data before loading it into a target system; ELT transforms it after loading. A streaming application instead processes incoming data event by event. It is worth considering when waiting for a scheduled batch would delay a useful action or insight. section 8.1.1 section 8.7
A processing layer can shape events for downstream systems:
- Filter or route records by content, rather than making producers publish every tailored variant or making each consumer discard irrelevant records. section 1.5.3
- Join streams, aggregate values, or use windows and maintained state when a result depends on more than one incoming event. section 1.5.3 section 8.7
- Publish the transformed result back to Kafka so another service can consume ready-made data. section 8.0
Enrich an order as it arrives
An application consumes order and customer events. Customer events keep a local state store current; when an order arrives, its customer identifier retrieves the corresponding information. The application emits an enriched order. Keeping state near the processing logic makes lookups fast, while synchronization with remote state supports recovery after failure. section 8.1
Figure 1.9 An example of a streaming application. RoutingService implements content-based routing, consuming messages from Addresses and, based on their contents (e.g., address type), publishing them to ShippingAddresses or BillingAddresses.
The routing diagram shows where content-based decisions can sit: between a raw address-event topic and separate downstream topics for shipping and billing addresses. section 1.5.3
Kafka brokers retain and deliver events; they do not inspect message content to perform application-specific routing. Kafka Streams and Apache Flink provide processing capabilities, while ksqlDB and managed services offer other implementation choices. Choose the processing layer and tool to match latency, transformation, and operational needs—not simply because data is available as events. Continuous, unbounded processing also makes ingestion, state, and output failures harder to diagnose than failures in a finite batch. section 1.5.3 section 8.7 section 8.5
