8 Designing Streaming Applications
This chapter covers
- An introduction to real-time processing and its key principles
- The design for building streaming applications
- The architecture of the Kafka Streams framework
- Exploring ksqlDB and Apache Flink for real-time data processing
The team gathered for their regular meeting, notebooks and laptops at the ready. Today’s topic quickly turned to the question of how to transform and aggregate their data as part of the Customer360 project.
Eva: Since we’re doing all this research, I think it’s worth considering transforming the data not just at the final service but somewhere in the middle. We could use a streaming framework for that.
Max: A streaming framework? What do you mean by that, Eva?
Eva: Let’s take our Customer360 Service as an example. Instead of doing the aggregation and joins in the final service that reads from Kafka, we could introduce an intermediate service. This service would join the data—like transactions and customer profiles—and produce an aggregated result back into Kafka. Then the final service just reads the ready-made aggregated data.
Max: What’s the point? I don’t see how the intermediate results would be useful to any other service.