7 Kafka Interaction Patterns

 

This chapter covers

  • Patterns for applying Kafka in microservices
  • Implementing a data mesh architecture with Kafka
  • Integrating data using Kafka Connect
  • An introduction to Kafka transactions

The team gathers once again in the meeting room, with the whiteboard cluttered with diagrams of their Kafka setup. Max Sellington, Eva Catalyst, and Rob Routine are now deep in discussion about how Kafka fits into their architecture.

Max: You know, one thing’s been bothering me. Not too long ago, service buses were all the rage. And they worked pretty well. Those tools let you visually define the data flow between services and even create transformations right there. But where are we now? It feels like we’ve taken a step back to low-level programming. Why? Is this all just for performance?

Eva: It’s not just about performance, Max. Those service buses never worked perfectly. The visual tools were useful, but they were never as expressive as code. Every time we needed to make a more complex change, it felt like we were fighting the tool’s limitations. And don’t forget, every small change often required redeploying the whole service bus pipeline, which wasn’t flexible at all. Flexibility is one of the main things we gain with Kafka and microservices.

7.1 Using Kafka in Microservices

7.1.1 Smart endpoints and dumb pipes

7.1.2 Request-response pattern

7.1.3 CQRS pattern

7.1.4 Event sourcing with snapshotting

7.1.5 Having “hot” and “cold” data

7.2 Implementing data mesh with Kafka

7.2.1 Motivations for data mesh

7.2.2 Domain ownership

7.2.3 Data as a product

7.2.4 Federated governance

7.2.5 Self-Serve platform

7.3 Using Kafka Connect

7.3.1 Kafka Connect at a glance

7.3.2 Internal Kafka Connect architecture

7.3.3 Converters

7.3.4 Single message transformations

7.3.5 Source connectors

7.3.6 Sink connectors

7.3.7 Changes in incoming data structure

7.3.8 Integrating Kafka and databases

7.3.9 Creating Connector for Customer 360

7.3.10 Common Kafka Connect problems

7.4 Ensuring delivery guarantee

7.4.1 Producer idempotence

7.4.2 Understanding Kafka transactions

7.4.3 Transactional outbox pattern

7.5 Online resources

7.6 Summary