Suppose one service needs shipping addresses while another needs billing addresses. Kafka brokers do not inspect message content, so putting the routing rule in a producer complicates upstream code, while repeating it in consumers makes each handle irrelevant events. A stream-processing application can instead read the address events and publish each type to its own topic. section 1.5.3
Kafka Streams is a Java library embedded in a standalone application, not a separate processing cluster. The application reads from Kafka topics, applies processing logic, and writes results to Kafka topics. Its API provides a high-level DSL for common operations; KStream represents a continuous event stream, while KTable represents a table-like view updated by incoming events. section 8.1.2 section 8.1.3
Choose the processing model around the work:
- Use DSL operations to filter, transform, join, or aggregate events. State stores support intermediate results, and windows support time-based aggregations such as hourly order counts. section 8.1.3 section 8.7
- Use the Processor API when predefined operations do not express the required logic, such as invoking an external anomaly-detection service. Custom processors can work with state stores and explicitly defined topology connections. section 8.3.3
Filtering orders
An orders application reads a raw-orders topic, keeps orders worth at least 50, maps each retained order to a smaller record containing its order and customer IDs, and writes to processed-orders. The application specifies the processing steps rather than manually wiring a consumer to a producer. section 8.1.3
Figure 8.9 Internally, the Kafka Streams application is represented as a graph. Events flow from input topics through source nodes, are processed by processing nodes, and are finally published to output topics via sink nodes.
The topology diagram shows the path from input-topic source nodes through processing nodes to output-topic sink nodes. DSL calls build this graph automatically; the Processor API allows its processors and connections to be specified explicitly. section 8.3.3
Deployment remains an architectural decision. Instances sharing a consumer group divide input partitions; partition counts limit processing parallelism, and failed instances trigger reassignment. Tasks carry partition assignments, topology instances, and any state stores. Kafka Streams supports exactly-once processing and backs stateful operations with changelog topics, but it does not remove the need to plan topics and partitions. section 8.1.2 section 8.3.4 section 8.7
Use streaming when results need to arrive with low latency, such as fraud alerts or a continuously enriched customer view. If daily processing suffices, batch ETL may be simpler and more cost-effective. Kafka Streams is a Java-centered choice; ksqlDB, Apache Flink, and managed services are alternatives to evaluate. section 5.1.4 section 8.0 section 8.7
