chapter twelve

12 Stream processing with Kafka Streams and ksqlDB

 

This chapter covers:

  • Getting started with Kafka Streams.
  • Using basic Kafka Streams APIs.
  • Using State Stores for persistent storage.
  • Enriching transaction streams.

So far in your journey in learning about Kafka, you’ve focused on the parts that made a complete event-streaming platform, including the Kafka brokers, producer clients, and consumer clients.

With this foundation, we can expand our toolset and understand the next layer of the Kafka ecosystem - stream processing using Kafka Streams and ksqlDB. These technologies offer abstractions, APIs, and DSLs based on the foundation that we have been building on in the previous chapters.

This chapter introduces a simple banking application - the processing of funds movement in and out of the accounts.

In our application, we will implement a Kafka Streams topology to process transaction requests submitted to the transaction-request topic atomically.

12.1 Kafka Streams

12.1.1 KStreams API DSL

12.1.2 KTable API

12.1.3 GlobalKTable API

12.1.4 Processor API

12.1.5 Kafka Streams Architecture

12.2 ksqlDB: An event streaming database

12.2.1 Queries

12.2.2 Local development

12.2.3 ksqlDB architecture

12.3 Going further

12.3.1 Kafka Improvement Proposals

12.3.2 Kafka projects you can explore

12.3.3 Community Slack Channel

12.4 Summary