liveBook · Manning

Processor topology

As explained in Kafka Streams in Action, Second Edition

A Kafka Streams topology is a directed graph of source, processor, and sink nodes that defines how records flow through a stream-processing application.

A processor topology is the Kafka Streams processing graph: nodes receive key-value records, transform or route them, and pass results onward. A minimal application has a source node that consumes a Kafka topic and a sink node that writes to another topic; processor nodes sit between them. section 6.0 section 6.7

Build the topology by reading a topic into a KStream, applying operations, and writing the resulting stream. A source is the graph root: it forwards consumed records into the application. Configure its key and value Serdes so Kafka Streams can deserialize broker-stored bytes into usable objects; configure produced records for the sink as well. section 6.3.1 section 6.7

A topology can take several useful shapes:

  • Read one topic, a collection of topics, or topics selected by a regular expression when identical processing should apply to each input. section 6.3.1
  • Forward one processor’s records to multiple downstream nodes, or feed one node from multiple inputs. section 6.6.3
  • Attach multiple sink nodes to the same processor to route records to different output topics. section 10.1.3

Uppercase stream

The Yelling app forms a three-node pipeline: consume strings from src-topic, map each value to uppercase, then produce the results to out-topic. More complex applications retain this shape while adding processing nodes and branches. section 6.3 section 6.3.1

Inspect the generated graph with TopologyDescription. Its arrows show downstream record flow and upstream lineage. Kafka Streams generates unique node names by default; assign names with Named, Consumed.withName, or Produced.withName so a printed topology communicates each node’s application role. A distinct source begins a sub-topology containing its downstream processors. section 6.6.3

Figure 6.2 Topology of the Yelling app
figure

Use the diagram to see the smallest complete topology: source input, uppercase processor, and sink output. section 6.3

How Kafka Streams in Action, Second Edition covers it

How deep it goes
Works through construction and inspection of node graphs, deferring sub-topology repartitioning details.
Examples
Uses an uppercase Yelling app and domestic-versus-international sales sinks to illustrate graph wiring.
What it assumes
Assumes Kafka topics and consumer records, while explaining byte deserialization through Serdes.
What you can do after
Build it from scratch by wiring sources, processors, and sinks through the DSL or Processor API.
How it connects
Connects Kafka Streams DSL, Processor API, KStream, Serdes, Kafka topics, TopologyDescription, and sub-topologies.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Kafka Streams in Action, Second Edition. The section links point to the parts of the book it draws on.