A central data team can become a bottleneck when every department’s data must pass through its ETL pipelines. If a business unit changes a data structure without notice, pipelines break. A data mesh shifts responsibility: departments publish usable data products, while a central team provides the platform and governance needed to share them. section 18.1.3 section 18.1.2
Four principles make that division of work practical:
- Domain ownership: each department supplies agreed-upon, high-quality data rather than waiting for a central team to collect it. section 18.1.2
- Data as a product: owners serve the needs of other departments or external partners. section 18.1.2
- Self-service infrastructure: teams can publish, consume, and modify data through a shared, domain-agnostic platform. section 18.1.2
- Decentralized governance: shared rules keep data documented, interoperable, secure, and compliant. section 18.1.2
When a department changes its data
In a centralized arrangement, an unannounced structural change can break the data team’s ETL pipeline. In a data mesh, the department that owns the data product is responsible for updating and documenting its schema. The platform still provides access and governance; ownership does not mean every department builds its own infrastructure. section 18.1.3 section 18.1.2
Kafka can serve as the shared platform for near-real-time exchange. Producing departments publish data products, manage their quality and life cycle, follow schema contracts, and control access. Consuming departments request access to the products they need. Kafka supports publishing and consumption, schema enforcement, and integration with governance tools; it does not replace the departments’ responsibility for useful data. section 18.1.4
Use Kafka for the flow of data products, not as a substitute for every storage or communication system. It is not a relational database for complex queries or current-state storage, nor a synchronous interface for immediate feedback. section 18.6
