A data format defines how data exchanged between software components is represented. In a Kafka event flow, producers and consumers need a shared understanding of message structure and serialization: Kafka brokers do not enforce either. A producer can write bytes that a consumer cannot interpret, so the agreement must be documented and implemented by the participating teams. section 2.4.2 section 3.1.4 section 6.5.1
Event design starts with a logical structure, then moves to a concrete format for implementation. That move requires decisions about supported types, nullable fields, nested structures, and how changes will affect consumers. A schema can serve as a language-independent contract for the event’s fields and rules; the serialization framework determines how the data is encoded and how the schema is used. section 6.2.1 section 6.3.1
Common choices have different architectural trade-offs:
- Avro uses a schema to produce compact binary data. Its schemas support fields, optionality, and nested records; data is not human-readable without interpretation. section 6.3.1
- Protobuf also uses a schema and compact binary data, with its own schema syntax. section 6.3.1
- JSON Schema describes JSON-based data. Both schema and data are human-readable, but the comparison identifies larger data size and less robust compatibility rules. section 6.3.1
Table 6.1 Comparison of different data formats
| Feature |
Avro |
JSON Schema |
Protobuf |
| Data serialization format |
Binary |
Text (JSON-based) |
Binary |
| Schema definition language |
JSON-based schema definition or Avro IDL (Interface Definition Language) |
JSON-based schema definition |
Protobuf-specific syntax |
| Compatibility |
Strong compatibility |
Less robust due to open content model |
Strong compatibility |
| Data size |
Compact |
Larger size |
Compact |
| Human readability |
Schema is human-readable; data is not |
Both schema and data are human-readable |
Schema is human-readable; data is not |
| Performance |
High |
Lower |
High |
| Support for complex types |
Supported |
Limited |
Supported |
| Programming language support |
Broad (Java, Python, |
Broad (any JSON-supporting language) |
Broad (Java, Python, C++, Go, etc.) |
Compare serialization, readability, data size, compatibility, and complex-type support before selecting a format. section 6.3.1
Choosing a format for a customer-data flow
A Customer 360 operational data store needs to exchange events across teams. JSON is familiar, but Avro is chosen for its tooling, community support, performance, and schema flexibility. The decision still needs a documented message structure and matching producer and consumer configuration. Topic keys are a separate architectural choice: a customer ID preserves the order of updates for that customer. section 6.3.2 section 2.4.2 section 3.1.4
The format agreement has operational consequences. A message with missing required fields or incompatible serialization may repeatedly stop a consumer at the same offset; brokers will not reject it on arrival. Test consumers against malformed messages, define a policy for handling them, and monitor any dead-letter topic used for review. In Kafka Connect, format-specific converters translate between Kafka data and ConnectRecord, while connectors handle external systems’ native formats. section 6.5.1 section 7.3.3
