8 Designing Streaming Applications

 

This chapter covers

  • An introduction to real-time processing and its key principles
  • The design for building streaming applications
  • The architecture of the Kafka Streams framework
  • Exploring ksqlDB and Apache Flink for real-time data processing

The team gathered for their regular meeting, notebooks and laptops at the ready. Today’s topic quickly turned to the question of how to transform and aggregate their data as part of the Customer360 project.

Eva: Since we’re doing all this research, I think it’s worth considering transforming the data not just at the final service but somewhere in the middle. We could use a streaming framework for that.

Max: A streaming framework? What do you mean by that, Eva?

Eva: Let’s take our Customer360 Service as an example. Instead of doing the aggregation and joins in the final service that reads from Kafka, we could introduce an intermediate service. This service would join the data—like transactions and customer profiles—and produce an aggregated result back into Kafka. Then the final service just reads the ready-made aggregated data.

Max: What’s the point? I don’t see how the intermediate results would be useful to any other service.

8.1 Introducing Kafka Streams

8.1.1 ETL, ELT and Stream Processing

8.1.2 Introduction to the Kafka Streams framework

8.1.3 Benefits of using Kafka Streams

8.2 Sketching out Customer360 with Kafka Streams

8.3 Processing data

8.3.1 Stateless Operations

8.3.2 Stateful operations

8.3.3 Processing API

8.3.4 Kafka Streams internal architecture

8.3.5 Windowing operations

8.3.6 Joining streams

8.3.7 Implementing CustomerJoinService

8.3.8 Interactive Queries

8.4 Alternative Solutions

8.4.1 Confluent ksqlDB

8.4.2 Apache Flink

8.4.3 Solutions from Cloud Providers

8.5 Common streaming application issues

8.5.1 Memory and disk capacity planning

8.5.2 Incorrect topic partitioning

8.5.3 Out-of-order data

8.5.4 Late arriving data

8.5.5 State Store initialization

8.5.6 Monitoring and debugging challenges

8.6 Online resources

8.7 Summary