liveBook · Manning

Stream-table join

As explained in Apache Kafka in Action

A stream-table join combines each incoming stream record with the table value associated with the record's key.

A stream-table join enriches each incoming stream record by looking up the value with the same key in a table. For example, sales data can be combined with the matching product record to add manufacturer information before the enriched event continues through the topology. section 12.4.3

The execution model has several practical consequences:

  • New data on the stream triggers the lookup and join immediately for that record. section 12.4.3
  • A table update does not join backward against earlier stream records; the updated value is used only when another stream record arrives. section 12.4.3
  • Matching depends on compatible keys and co-partitioning: the inputs need the same key and the same number of partitions. section 12.4.3
  • Use a table-table join when changes on either side must trigger execution; use a stream-stream join with time windows when both inputs are infinite streams. section 12.4.3 section 12.7

Enriching sales with product master data

Read salesData as a stream and products as a table. The join function receives a sale, looks up its product key, copies the product manufacturer into the sale, and emits the enriched record. A later product update changes future enrichments but does not reprocess existing sales automatically. section 12.4.3

KStream salesData = builder.stream("salesData");    #1
KTable masterData = builder.table("products");    #2
ValueJoiner enricher = (salesData, masterData) -> {    #3
    salesData.manufacturer = masterData.manufacturer;
    return salesData;
};
KStream salesData_enriched = salesData.join(
    masterData, enricher);    #4

Inspect the Kafka Streams topology pattern: create a KStream for sales, create a KTable for products, define a ValueJoiner, and call join. section 12.4.3

How Apache Kafka in Action covers it

How deep it goes
Explains keyed lookup, trigger direction, and co-partitioning, while stopping short of detailed state-store internals.
Examples
Uses salesData with products, plus loyalty cards with customers, to demonstrate enrichment and contrasting join triggers.
What it assumes
Assumes keyed stream and table inputs, then explains why matching keys and co-partitioning are required.
What you can do after
Apply it via Kafka Streams by joining a KStream with a KTable and supplying a ValueJoiner.
How it connects
Connects stream processing with tables, co-partitioning, table-table joins, stream-stream joins, and time windows.

Also explained in

Compare how 2 books explain stream-table join →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.