Suppose records arrive in Kafka and another application needs a filtered or transformed stream. Kafka Streams is a Kafka-native Java library for that processing. A Streams application acts as both consumer and producer: it reads records, processes them, and writes results back to Kafka. It can filter, transform, and merge data rather than requiring a hand-built combination of consumers and producers. section 4.3.3 section 12.1.1
That read–process–write pattern raises a reliability question: output may span topic partitions, while progress is recorded in consumer offsets. Those writes need to be coordinated for exactly-once processing. Kafka Streams provides a library for this common pattern; merely writing a consumer and producer does not resolve the coordination problem. section 5.2.1
Filtering a 12-partition topic
Kafka Streams creates one task per partition, so filtering a 12-partition topic creates 12 tasks. One application instance handles all 12; with six instances, each typically handles two. Adding instances distributes existing tasks rather than creating more of them. If an instance fails, Kafka’s rebalance protocol assigns its tasks to surviving instances. section 12.6
Figure 12.13 Kafka Streams applications are divided into tasks that are distributed across different threads on different instances. If an instance fails, another instance takes over the tasks.
See how tasks are distributed across instance threads and reassigned after an instance fails. section 12.6
For production use, keep the work and its operating cost in view:
- Tasks are the smallest processing units, and their count depends on the partitions being processed. Tasks can run across instances and across threads within an instance; configure the latter with num.threads. section 12.6
- Monitor failed threads, processing rates and duration, and dropped records. Repeated task termination and recreation outside startup, shutdown, or topology changes calls for investigation. section 15.3.4
- Avoid assuming every transformation belongs in Streams. Consolidating raw core-system data may require so many joins and state stores that the application becomes difficult to operate; complex joins in a separate relational database, followed by CDC, may be preferable. section 18.2
