A Kafka topic might carry data that several services need to read. Kafka stores message content as byte arrays; it does not interpret the contents or require a particular data format. Producers and consumers therefore need an agreed way to write and interpret each message. section 3.2.2 section 17.2.1
Choose a format for the people and systems that will use the data:
- JSON is easy for people and machines to read, but its size can grow substantially. XML has a similar size concern. section 3.2.2
- Protobuf and Avro are options when minimizing size matters. Reading binary data by hand is more cumbersome, and interpreting it requires a schema. section 3.2.2
- Keep format choices consistent across topics and document them. Schemas also help maintain uniformity for JSON and XML, not just binary formats. section 3.2.2
The producer serializes a message’s key and value into byte arrays before sending them. Kafka libraries commonly provide serializers for JSON, Protobuf, and Avro; other formats can use custom serializers and deserializers. Serialization is separate from partition selection, which happens once the key and value are available as bytes. section 8.1.2
Core-system data is not necessarily a shared format
Debezium can use change data capture to put changes from a core system’s database into Kafka. That makes the data available, but it also carries the core system’s format, which other teams may struggle to interpret. Rather than asking every application to reuse normalized data spread across many tables, a pipeline can publish consolidated structures for its consumers. Such a transformation may require complex joins and state stores, so its operating cost matters too. section 18.2
