liveBook · Manning

Foreign key association

As explained in Kafka Streams in Action, Second Edition

A KTable foreign key join combines table records by a foreign key derived from the joining record rather than requiring identical primary keys.

A foreign key association joins two KTables when their primary keys differ but a field in one table’s value identifies a record in the other table. It fits records such as client contracts keyed by client ID whose values carry a commodity code. Unlike a primary-key table join, the association is derived from data in the left-side value. section 8.6.4 section 8.7

The join uses a function to extract the lookup key from the left value, then combines matching values with a joiner. A null extracted key produces no join. Inner and left-outer table joins support this form, and updates on either side can trigger a new result. section 8.6.4

Build the association in three parts:

  • Keep the left table keyed by its own identity, even when the field needed for association is only inside its value. section 8.6.4
  • Provide the right KTable, a foreign-key extractor that returns its key, and a joiner to KTable.join. section 8.6.4
  • Choose inner semantics when a match is required, or left-outer semantics when unmatched left records must remain visible. section 8.6.4
  • Do not impose co-partitioning solely for this association: the lookup targets a value-derived key rather than the tables’ original keys. section 8.6.4 section 11.6

Activity counts and rewards members

An activity-count table keyed by last_name can join a rewards-members table keyed by member_id through the members table’s last_name column. The resulting table connects fitness participation to rewards-member information even though the tables use different primary keys. section 11.6

Listing 8.13 KTable foreign key join
clientContractTable.join(commodityPriceTable,      #1
               foreignKeyExtractor,    #2
               joiner);    #3

Use this minimal KTable API shape: the extra argument is the function that extracts the right-table key from each left-side value. section 8.6.4

Internally, the association needs more than a direct local lookup. The processing flow extracts the foreign key, sends it through repartition topics, performs a right-table lookup, and checks the original left value by hash before producing the result. A right-side update can scan stored composite keys to emit all affected matches. section 8.6.4

How Kafka Streams in Action, Second Edition covers it

How deep it goes
Explains extractor-based joins and update propagation down to repartition topics, lookups, hashes, and composite-key state scanning.
Examples
Uses client contracts and commodity prices, then joins activity counts with rewards members through last_name.
What it assumes
Assumes familiarity with KTables, keys, partitions, and table-table joins while distinguishing value-derived association keys.
What you can do after
Apply it via KTable.join or ksqlDB table joins by deriving an association key from a table value.
How it connects
Connects KTable joins, primary keys, co-partitioning, repartition topics, state stores, and inner and left-outer joins.
Cover of Kafka Streams in Action, Second Edition
liveBookmanning.com
Read Kafka Streams in Action, Second Edition on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Kafka Streams in Action, Second Edition. The section links point to the parts of the book it draws on.