Copartitioning

As explained in Kafka for Architects

Copartitioning is the arrangement of multiple input topics with matching partition counts and partitioning logic so records with corresponding keys are processed together.

Copartitioning arranges input topics for reliable stream joins: every topic uses the same partition count, key distribution strategy, and partitioning logic. Records with the join key then reach partitions with the same index and are processed by the same Kafka Streams task. section 8.3.6

Orders and payments

To join orders with payments, both topics must use the same number of partitions and the same key, such as orderId. If payments use paymentId, selectKey can change the payment key to orderId before joining. section 8.3.6

Design checks for copartitioned topics:

• Match the partition count across all input topics. section 8.3.6
• Use the join attribute as the key and apply the same partitioner. section 8.3.6
• Treat key-distribution correctness as an application responsibility because Kafka Streams does not validate it at runtime. section 8.3.6

Figure 8.14 When the Payments and Orders topics are joined, they must have the same number of input partitions and the same key (orderId).

Shows the aligned partition counts and shared orderId key required when joining Payments and Orders. section 8.3.6

How Kafka for Architects covers it

How deep it goes

Defines partition, key, and partitioner requirements and assigns correctness responsibility to programmers.

Examples

Uses Orders and Payments topics, orderId alignment, paymentId replacement, and same-index task assignment.

What it assumes

Assumes familiarity with topics, partitions, keys, and stream joins.

What you can do after

Configure copartitioned Kafka Streams inputs and rekey records before joining them.

How it connects

Connects stream joins with partitions, keys, partitioners, Kafka Streams tasks, and selectKey.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Kafka Streams in Action, Second EditionCo-partitioning is the arrangement in which participating Kafka Streams inputs have the same partition count and compatible key types so corresponding records can be processed together.

Compare how 2 books explain copartitioning →

Related concepts

• joinused by · A join is an operation that combines records or rows from multiple data sources according to specified matching conditions or relationships.
• stream-global-table joincontrasts with · A stream-global-table join combines records from a KStream with matching records in a fully replicated GlobalKTable.
• GlobalKTablecontrasts with · GlobalKTable is a Kafka Streams abstraction that replicates all records from an underlying topic to every application instance for read-only access.
• foreign key associationcontrasts with · A foreign key association is a relationship or join between records whose key in one entity or table references a key in another.
• StreamPartitioneruses · StreamPartitioner is a Kafka Streams interface for determining the output partition assigned to a record.
• repartitionproduced by · Repartitioning is the redistribution of records or items across partitions according to a partitioning rule, potentially changing the number of partitions.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.