Stream partitioning

As explained in Kafka for Architects

Stream partitioning is the logical distribution of stream records among parallel processing tasks during execution.

Stream partitioning determines how records are distributed among parallel processing tasks. In Kafka Streams, partitioning by key follows the underlying Kafka topic partitions. In Apache Flink, it is a logical distribution during execution, independent of Kafka or another external system. section 8.4.2

Orders by region

To count orders per region, use region as the key and group events accordingly. Flink’s keyed state associates the resulting state with each key, while its execution plan and task parallelism determine how work is distributed. section 8.4.2

Figure 8.21 The Flink architecture. The Flink Client is responsible for transforming a program into a graph, which is submitted to the JobManager. JobManager converts it into an ExecutionGraph and deploys tasks to available TaskManagers.

The architecture diagram shows how an ExecutionGraph deploys tasks to TaskManagers, providing context for Flink’s task-level distribution. section 8.4.2

How Kafka for Architects covers it

How deep it goes

Defines Flink’s logical partitioning by contrast with Kafka Streams and a keyed aggregation example, without detailing partitioning algorithms.

Examples

Illustrates grouping orders by region to calculate per-region counts.

What you can do after

Evaluate/understand how Flink distributes keyed records independently of Kafka topic partitions.

How it connects

Connects partitioning to Kafka topic partitions, Kafka keys, task parallelism, aggregation, and keyed state.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.