Kafka Streams is Kafka’s native Java stream-processing library for client applications. It runs outside Kafka brokers, connects to a Kafka cluster, and lets applications transform event records, including with stateful operations such as joins and aggregations. section 1.4.5 section 1.6
A Kafka Streams application is a processing topology: a graph whose source nodes read topic records, processing nodes transform key-value records, and sink nodes write results to Kafka topics. The API abstracts over Kafka producers and consumers so the application can focus on stream-processing requirements. section 6.0 section 6.1
Figure 6.1 Kafka Streams is a graph with a source node, any number of processing nodes, and a sink node.
Use the topology diagram to identify the minimum source-to-sink flow and where additional processing nodes fit. section 6.1
A typical application is assembled in this order:
- Set required application.id and bootstrap.servers properties. The application ID identifies the logical application; bootstrap servers establish the cluster connection. section 6.3.2
- Create Serdes for record serialization and deserialization, then build the processor topology. section 6.3
- Create and start Kafka Streams. Applications sharing an application.id are treated as one logical application. section 6.3 section 6.7
- Transform, select, split, merge, or inspect records with operations such as mapValues, map, filter, branch, merge, selectKey, peek, and print. section 6.7
From incoming text to retail decisions
The minimal Yelling app reads text, converts values to uppercase, and writes them to an output topic. A more realistic retail topology masks purchase-card digits, emits purchase and regional data to topics, and supplies reward data for downstream consumers. Another real-time application builds location-based purchase trends and writes them to a sales-trends topic for dashboard export. section 6.3 section 6.4 section 1.5
Spring Boot can manage a Kafka Streams application with @EnableKafkaStreams. It creates lifecycle support around KafkaStreams; configuration supplies the streams settings, while a StreamsBuilder defines the topology. Customizers can attach a state listener, and explicit startup and shutdown can be handled with KafkaStreams.start and close. section 12.3
