Subtopology

As explained in Apache Kafka in Action

A subtopology is a portion of a Kafka Streams processing topology separated to write to or read from an intermediate topic.

A subtopology is one part of a Kafka Streams processing topology, separated from another part by an intermediate topic. This split lets processing continue after data is written to and read back from a repartition topic. section 12.6

Kafka Streams needs this boundary when processing must be coordinated by key, such as during a join or aggregation. The first subtopology writes to an internal topic ending in -repartition; the second reads from it and continues processing. The design improves correctness but increases network traffic and storage requirements. section 12.6 section 12.7

Changing a key before counting

A topology that calls selectKey() and then count() changes the record key before aggregation. Kafka Streams therefore repartitions the data, creating one subtopology for selectKey() and another for reading the repartitioned data and performing the count. section 12.6

Figure 12.14 If we first execute a selectKey() and then an aggregation such as count(), the topology needs to be split up into multiple subtopologies.

Use the topology diagram to trace how selectKey() followed by an aggregation divides processing into multiple subtopologies. section 12.6

How Apache Kafka in Action covers it

How deep it goes

Explains the split mechanism, repartition-topic flow, selectKey() example, and network and storage costs.

Examples

Demonstrates selectKey() followed by count(), separated by an internal -repartition topic.

What you can do after

Evaluate/understand why Kafka Streams splits key-changing processing before joins or aggregations.

How it connects

Connects subtopologies with Kafka Streams, topology, repartitioning, intermediate topics, keys, joins, aggregations, tasks, and partitions.

liveBookmanning.com

Read Apache Kafka in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Kafka Streams in Action, Second EditionA sub-topology is a portion of a Kafka Streams topology containing a source node and every processor downstream of that source.

Compare how 2 books explain subtopology →

Related concepts

• processor topologypart of · A processor topology is a directed graph of source, processor, and sink nodes that defines how records flow through a stream-processing application.
• repartitionproduced by · Repartitioning is the redistribution of records or items across partitions according to a partitioning rule, potentially changing the number of partitions.
• source pointhas part · A source node is a designated origin from which data, flow, traversal, or directed relationships proceed.

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.