6 Defining Data Contracts

 

This chapter covers

  • Event design principles
  • Supporting data contracts in Kafka
  • Type evolution and schema changes
  • Common challenges in managing data contracts

The team gathers to discuss an important question about how to define data contracts.

Max: Alright team, we haven't really talked about events yet. What exactly is an event, and how do we define it when systems communicate through Kafka? It's not like a service with a clear endpoint and parameters. So, how does a consumer know what data to expect?

Eva (nodding): Good question, Max. Let’s break it down. An event is a record of something that has happened in the system. It’s immutable, meaning it can’t be changed once created, and it includes a timestamp to show when it occurred. The context of the event gives it meaning and relevance. In Kafka, you define the schema for the events. The schema acts as the contract between the producer and the consumer. It specifies the structure of the data being exchanged. This is the most important topic for us as architects because it ensures consistency and reliability across our systems.

Max: Got it. So, are these schemas defined in XML?

Rob (laughing): XML? Nobody uses XML anymore, Max. That went out with dial-up internet. We use more modern formats like Avro, JSON Schema, and Protobuf. But if you're feeling nostalgic, you could always define custom serializers.

6.1 How Kafka handles event structure

6.2 Designing events

6.2.1 Challenges in event design

6.2.2 Fact and delta events: representing state changes

6.2.3 Composite, atomic, and aggregate events: representing event structure

6.2.4 Pulling state on notification

6.2.5 Evolution of types

6.2.6 Mapping events to Kafka messages

6.2.7 Data strategies for Customer 360 project

6.3 Event Governance

6.3.1 Data Formats

6.3.2 Selecting data format for Customer 360 project

6.3.3 Data Ownership

6.3.4 Organizing data and communicating changes

6.3.5 Designing events for Customer 360 project

6.4 Schema Registry

6.4.1 Schema Registry in Kafka ecosystem

6.4.2 Registering schemas

6.4.3 A Concept of Subject

6.4.4 Compatibility Rules

6.4.5 Alternatives to Schema Registry

6.4.6 Handling data contracts without the centralized server

6.4.7 Commercial extensions for data contracts

6.5 Common problems in handling data contracts

6.5.1 Absence of server-side validation

6.5.2 Handling incompatible changes for non-compacted topics

6.5.3 Migrating state

6.5.4 Automatic registration of schemas

6.6 Online Resources

6.7 Summary