12 Stream processing with Kafka Streams and ksqlDB

 

This chapter covers

  • Getting started with Kafka Streams
  • Using basic Kafka Streams APIs
  • Using state stores for persistent storage
  • Enriching transaction streams

So far on our path in learning about Kafka, we’ve focused on the parts that help make a complete event-streaming platform, including the Kafka brokers, producer clients, and consumer clients. With this foundation, we can expand our toolset and understand the next layer of the Kafka ecosystem—stream processing using Kafka Streams and ksqlDB. These technologies offer abstractions, APIs, and DSLs (domain-specific languages), based on the foundation that we have built on in the previous chapters.

This chapter introduces a simple banking application that processes funds as they move in and out of the accounts. In our application, we will implement a Kafka Streams topology to process the transaction requests submitted to the transaction-request topic atomically.

NOTE

Our business requirement states that we must check whether the funds are sufficient for every request received before updating the account’s balance that’s being processed. As per our requirements, our application can’t process two transactions simultaneously for the same account, which could create a race condition in which we cannot guarantee we can enforce the balance check before withdrawing funds.

12.1 Kafka Streams

12.1.1 KStreams API DSL

12.1.2 KTable API

12.1.3 GlobalKTable API

12.1.4 Processor API

12.1.5 Kafka Streams setup

12.2 ksqlDB: An event-streaming database

12.2.1 Queries

12.2.2 Local development

12.2.3 ksqlDB architecture

12.3 Going further

12.3.1 Kafka Improvement Proposals (KIPs)

12.3.2 Kafka projects you can explore

12.3.3 Community Slack channel

Summary

References