Schema registry

As explained in Kafka for Architects

A schema registry is a service that centrally stores, versions, and validates schemas for structured data exchanged between producers and consumers.

Kafka brokers store messages as bytes; they do not inspect a message or enforce its structure. A schema registry supplies the missing shared contract. It stores formal definitions of event data separately from brokers so producers and consumers can agree on message structure without a direct dependency on each other. section 1.5.1

The contract travels by identifier, not by copying the schema into every message:

• A schema is registered and assigned an ID. Changes create new, immutable versions, which clients can cache. section 1.5.1
• A configured producer serializes the data and includes the schema ID in the message. A configured consumer uses that ID to retrieve the schema and deserialize the data. section 6.4.1
• Compatibility checks help govern changes between versions, but they do not make Kafka brokers validators: clients remain responsible for using schemas correctly. section 6.7

Figure 6.12 Interactions with the Schema Registry. The schema is registered with the Schema Registry server, which stores it in Kafka. Producers use this schema to serialize messages, and consumers retrieve the schema from the Schema Registry to deserialize messages.

The interaction diagram separates schema registration and retrieval from Kafka message delivery: the registry stores schemas in Kafka, while producers and consumers use them to serialize and deserialize events. section 6.4.1

Operationally, the registry exposes an HTTP REST API for registering and retrieving schemas and checking compatibility. It runs as a cluster for failover and persists schema data in a compacted Kafka topic named _schemas by default. Supported formats here are Avro, JSON Schema, and Protobuf; clients need matching format-aware serializers or deserializers and a registry URL. section 6.1 section 6.4.1

Choosing a registration path

A producer can register a compatible new version automatically, or a team can register it through the REST API before release. Manual registration better fits a controlled release process. Confluent Platform clients can disable automatic registration with auto.register.schemas=false and configure use.latest.version=true to look up the latest version in a subject. section 6.4.2

Central management does not eliminate migration work. Renaming fields, changing types, or restructuring entities may exceed a compatibility mode and require a planned transition. Schema IDs can also differ between environments, complicating transfer between Kafka clusters. Distributing schemas and custom serializers in client libraries is an alternative, but it shifts coordination to clients and can limit integration with message-browsing tools. section 6.4.6

How Kafka for Architects covers it

How deep it goes

Traces registration, schema IDs, client serialization, REST access, storage, and release choices, while treating validation as a client responsibility.

Examples

Uses a producer-to-consumer interaction diagram, REST requests, Avro client configuration, and automatic-versus-manual registration choices.

What it assumes

Assumes familiarity with Kafka producers, consumers, and events; defines why broker opacity calls for a separate schema service.

What you can do after

Operate/configure it by selecting registration controls, configuring format-aware clients, and planning migrations when compatibility rules are insufficient.

How it connects

Connects data contracts and schema evolution to Kafka producers, consumers, serializers, compatibility checks, and client-side validation.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Kafka Streams in Action, Second EditionSchema Registry is a service for storing and validating schemas so that serialized records remain compatible between producers and consumers.
• 

  liveBookmanning.com
  Apache Kafka in ActionA schema registry is a service that stores, versions, and provides access to schemas governing data formats.

Compare how 3 books explain schema registry →

Related concepts

• RESTuses · REST is an architectural style for distributed hypermedia systems that organizes interactions around resources and representations through constraints including a uniform interface, stateless communication, cacheability, and layered components.
• Gradle pluginused by · A Gradle plugin is an extension that adds project configuration, tasks, or integrations to the Gradle build system.
• backward compatibilityuses · Backward compatibility is the property of a changed or newer system remaining usable by existing clients and consumers of its earlier interfaces, behavior, or data.
• schemauses · A data schema is a formal specification of the structure, fields, types, relationships, and constraints of data.
• Avrouses · Apache Avro is a language-independent data serialization system that uses schemas to encode structured data, commonly in compact binary or JSON form.
• Apache Kafkauses · Apache Kafka is a distributed event-streaming platform for publishing, storing, processing, and consuming streams of records.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.