Chapter 6. Schemas

In the first part of this book, we took a wide-ranging look at event streams and the unified log, using fictitious online retailer Nile. We looked in depth at adding a unified log to our organization and experimented with different stream-processing frameworks to work with the events in our Kafka topics.

But like fast-food addicts, we didn’t spend a lot of time thinking about the quality of the events that we were feeding into our unified log. This part of the book aims to change this, by looking much more closely at the way we model the events flowing through our unified log, using schemas.

Working for Plum, a fictitious global consumer-electronics manufacturer, we will introduce Plum’s first event, a regular health-check “ping” emitted from each NCX-10 machine on the factory floor. Like every unified log, Plum’s is fundamentally decoupled: consumers and producers of event streams have no particular knowledge of each other. This puts the onus on event schemas to serve as the contract between Plum’s event consumers and producers.

6.1. An introduction to schemas

6.2. Modeling our event in Avro

6.3. Associating events with their schemas

Summary