Master data is relatively stable, authoritative information about core business entities such as customers and products. Unlike transactional data, which records events such as price changes, it describes the entities those events refer to. section 12.1
In stream processing, master data commonly acts as a table: a compacted topic stores the current value for each entity key. A sales stream can use the product ID to look up current product information and enrich each sale. section 12.4.3
A practical master-data lookup has three parts:
- Store relatively infrequently changing entity records in a compacted topic. section 12.4.3
- Key the table by the entity identifier, such as product ID. section 12.4.3
- Look up the current record whenever a stream event arrives, then combine both values into an enriched result. section 12.4.3
Enriching sales with product information
A sales stream keyed by product ID can join a product master-data table to add the product name or manufacturer before monitoring displays the sale. The join runs when a sales message arrives and returns another stream. section 12.4.3
KStream salesData = builder.stream("salesData"); #1
KTable masterData = builder.table("products"); #2
ValueJoiner enricher = (salesData, masterData) -> { #3
salesData.manufacturer = masterData.manufacturer;
return salesData;
};
KStream salesData_enriched = salesData.join(
masterData, enricher); #4
Shows the Kafka Streams topology that creates a sales stream, reads product master data as a table, and applies an enrichment join. section 12.4.3
