A customer-profile update may be useful to billing, reporting, and delivery systems, but each consumer needs to know what the event contains. A schema is the formal description of a message’s structure: its fields, types, and constraints. It makes that structure a data contract between producer and consumer. section 6.2.6 section 1.4.1
Kafka brokers store and forward messages as opaque bytes; they do not enforce a schema. Producers and consumers therefore need an agreement outside the broker. With Schema Registry, a producer uses a registered schema and places its identifier in the message. A consumer uses the identifier to retrieve the schema and deserialize the data. Clients can cache immutable schemas. section 1.5.1 section 6.7
Figure 6.1 Managing schemas in Kafka. Schemas are deployed via a REST API and stored in a Kafka topic. The Schema Registry provides access to these schemas through its API, allowing clients to retrieve and cache the necessary schemas locally.
The diagram traces schema registration, storage, and client retrieval separately from the Kafka messages that carry schema identifiers. section 6.1
Design the contract before choosing its serialization format:
- Choose the event boundary. A fact event carries an entity’s full state, simplifying a new consumer’s view; a delta event carries only the change, reducing message size but increasing consumer complexity. section 6.2.2 section 6.7
- Decide whether timestamps, versions, and sources belong in the event structure or separate metadata. Translate logical definitions into concrete types, optional fields, and nested structures deliberately; that translation is not automatic. section 6.2.1
- Choose Avro, Protobuf, or JSON Schema according to the required data representation and compatibility behavior, rather than assuming one format fits every event. section 6.3.1
Listing 6.3 An example of the Avro format
{
"type": "record", #1
"name": "Client",
"namespace": "com.example", #2
"fields": [
{ "name": "clientId", "type": "string" }, #3
{ "name": "clientName", "type": "string" },
{ "name": "age", "type": "int" },
{ "name": "email", "type": ["null", "string"], "default": null }, #4
{ "name": "isPremium", "type": "boolean", "default": false },
{ "name": "address",
"type": {
"type": "record", #5
"name": "Address",
"fields": [
{ "name": "street", "type": "string" },
{ "name": "city", "type": "string" },
{ "name": "postalCode", "type": "string" }
]
}
},
{ "name": "createdAt", "type": "long", "logicalType": "timestamp-millis" }
]
}
The Client Avro schema makes fields, primitive types, an optional email, defaults, and a nested address concrete. section 6.3.1
Schema ownership matters as much as syntax. Keep contracts accessible in a central, version-controlled repository managed by the producer team. Notify consumers of new versions and give them a way to propose changes. Schema Registry versions registered changes and checks compatibility, but breaking changes can still require phased deployments or transformations. section 6.3.4 section 6.7
From customer-profile model to deployable schema
For a Customer 360 proof of concept, full-state profile events suit a compacted topic. One design defines their structure abstractly in AsyncAPI; another stores Avro schemas in Git and references them from AsyncAPI. The latter avoids potential gaps when translating an abstract model into the concrete schema developers use. section 6.3.5
