6 Defining Data Contracts

 

This chapter covers

  • Event design principles
  • Supporting data contracts in Kafka
  • Type evolution and schema changes
  • Common challenges in managing data contracts

With the fundamentals of Kafka and its real-world fit in place, let’s now turn to a practical question: how to define events and express them as durable, evolvable data contracts. We’ll set event boundaries and granularity, compare state versus delta events (and when each fits), and map domain events to Kafka messages without losing business meaning. You’ll learn the tools and techniques for managing contracts over time: type evolution and compatibility, format choices, ownership, and how teams communicate changes. Finally, we make Schema Registry concrete—its architecture, the idea of subjects, compatibility rules—and survey practical alternatives. This should set you up with a clear path from whiteboard models to production-ready schemas that teams can rely on.

6.1 Field notes: Turning business facts into schemas

The team gathers to discuss an important question about how to define data contracts.

Max: Alright team, we haven't really talked about events yet. What exactly is an event, and how do we define it when systems communicate through Kafka? It's not like a service with a clear endpoint and parameters. So, how does a consumer know what data to expect?

6.2 How Kafka handles event structure

6.3 Designing events

6.3.1 Challenges in event design

6.3.2 Fact and delta events: representing state changes

6.3.3 Composite, atomic, and aggregate events: representing event structure

6.3.4 Pulling state on notification

6.3.5 Evolution of types

6.3.6 Mapping events to Kafka messages

6.3.7 Field notes: Data strategies for Customer 360 project

6.4 Event Governance

6.4.1 Data Formats

6.4.2 Field notes: selecting data format for Customer 360 project

6.4.3 Data ownership

6.4.4 Organizing data and communicating changes

6.4.5 Field notes: Designing events for Customer 360 project

6.5 Schema Registry

6.5.1 Schema Registry in Kafka ecosystem

6.5.2 Registering schemas

6.5.3 A Concept of Subject

6.5.4 Compatibility Rules

6.5.5 Alternatives to Schema Registry

6.5.6 Handling data contracts without the centralized server

6.5.7 Commercial extensions for data contracts

6.6 Common problems in handling data contracts

6.6.1 Absence of server-side validation

6.6.2 Handling incompatible changes for non-compacted topics

6.6.3 Migrating state

6.6.4 Automatic registration of schemas

6.7 Online Resources

6.8 Summary