liveBook · Manning

Self-service data infrastructure

As explained in Apache Kafka in Action

Self-service data infrastructure is shared technical infrastructure that lets domain teams independently publish, consume, modify, and operate data capabilities without case-by-case intervention from a central data team.

Self-service data infrastructure gives domain teams a shared, domain-agnostic platform for publishing, consuming, and modifying data without lengthy central-team requests. The platform enforces agreed processes and data-protection rules while shifting routine data ownership toward the departments that understand the data. section 18.1.2

A practical division of responsibility looks like this:

  • Domain teams deliver high-quality data products and satisfy the needs of consuming departments or external partners. section 18.1.2
  • The data team provides technical support, governance guidance, reusable producer and consumer templates, and help with significant schema changes. section 18.1.3
  • Kafka can provide a central hub for real-time exchange while centralized clusters simplify management and support self-service tools. section 18.6

From handoffs to enablement

In a traditional arrangement, a central data team collects data from every service into a data lake, often without enough domain knowledge to assess its quality. Self-service infrastructure moves routine publishing and consumption to domain teams, allowing the central team to become a center of excellence rather than continually repairing broken pipelines. section 18.1.3

Figure 18.1 The core principles of a data mesh
The core principles of a data mesh

Use the data-mesh principles diagram to relate self-service infrastructure to domain ownership, data as a product, and decentralized governance. section 18.1.2

How Apache Kafka in Action covers it

How deep it goes
Defines the operating model and responsibilities, but stops short of detailed platform implementation or configuration procedures.
Examples
Contrasts centralized data lakes and data swamps with producer-consumer templates, schema-evolution support, and Kafka-based exchange.
What it assumes
Assumes organizational domains and a supporting data team, while treating domain ownership and governance as connected operating principles.
What you can do after
Operate within a shared platform to publish, consume, and modify data while following data-protection rules.
How it connects
Connects data mesh, domain ownership, data as a product, decentralized governance, Kafka, and schema evolution.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.