How deeply does Kafka for Architects cover reference table?

Contrasts snapshot replication with CDC and identifies connector choices, but stops short of implementation configuration.

Reference table

As explained in Kafka for Architects

A reference table is a relatively stable table of lookup or classification values shared by other data-processing systems.

A reference table stores relatively stable lookup or classification values, such as marital status, occupations, education levels, and country codes. Replicating these tables usually needs less application-specific logic than replicating profile changes. section 5.1.2

Choose the replication approach according to how the source data changes:

Use snapshot-based replication to transfer the source’s current state. section 5.1.2
Use change data capture (CDC) to stream inserts, updates, and deletions from source-system change logs. section 5.1.2
Use Kafka Connect configurations and connectors instead of custom update-notification services; JDBC Source Connector supports snapshots, while Debezium can provide CDC. section 5.1.2

Replicating lookup data

Source connectors can pull reference data from PostgreSQL and MongoDB into Kafka topics. Sink connectors can then read those topics and populate an MS SQL database and Amazon S3.

How Kafka for Architects covers it

liveBook manning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Related concepts

contextkind of · Context is the surrounding information, state, or circumstances that condition how an entity, event, or task is interpreted or how a system behaves.
GlobalKTableused by · GlobalKTable is a Kafka Streams abstraction that replicates all records from an underlying topic to every application instance for read-only access.
data enrichmentproduces · Data enrichment is the process of augmenting existing data with additional attributes, context, relationships, or derived information.
data testingused by · Data testing is the systematic verification of data against expected correctness, completeness, consistency, validity, and other quality requirements.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.