Schema validation checks data against a defined structure and its rules. In Kafka, records are byte arrays, so validation must happen outside the basic broker path or through additional platform capabilities. section 13.1.3
Choose the enforcement point according to security, ownership, and operational needs:
- Restrict writes through the Single Producer Pattern, allowing only the topic owner’s team to publish. This improves security but limits Kafka’s flexibility. section 13.1.3
- Validate at the broker when using Confluent Server or Confluent Cloud, which provide native broker-side schema validation. Kafka itself does not implement this capability. section 13.1.3
- Insert a Kafka proxy when broker-side validation is unavailable. Conduktor provides a commercial option, while Kroxylicious provides an open source alternative backed by Red Hat. section 13.1.3
- Manage schema versions through the deployment pipeline rather than allowing producers to create versions automatically. This improves traceability and reduces unintended compatibility problems. section 13.1.3
A governed data product
A producing department publishes a data product using an agreed schema contract. Kafka helps enforce the predefined structure, while consuming departments ensure their software remains compatible with the contract and related governance policies. section 18.1.4
Figure 18.3 Kafka in a data mesh architecture
Use the architecture figure to trace how schema enforcement fits alongside data publishing, consumption, and governance in a data mesh. section 18.1.4
