Network topology

As explained in Kafka for Architects

A graph topology is a directed processing structure that defines how streaming records traverse processing steps.

A network topology is a directed acyclic graph that defines how records move from Kafka source topics through processing steps to output topics. It makes a streaming application’s data flow explicit. section 8.3.3

A topology connects three kinds of nodes:

• Source nodes read records from input topics. section 8.3.3
• Processing nodes filter, transform, aggregate, or otherwise handle records; stateful operations can use linked state stores. section 8.3.3
• Sink nodes publish processed records to output topics. Edges define the flow between nodes, in memory or through Kafka topics. section 8.3.3

Kafka Streams builds the graph automatically when application logic uses DSL functions. For custom logic, such as calling an external anomaly-detection service, developers can define the topology programmatically and implement reusable Java processors. These can access state stores, schedule actions, or use the Kafka client API. section 8.3.3

Figure 8.9 Internally, the Kafka Streams application is represented as a graph. Events flow from input topics through source nodes, are processed by processing nodes, and are finally published to output topics via sink nodes.

Follow the path from input topics through source and processing nodes to output topics. section 8.3.3

At runtime, each Kafka Streams task combines an instance of the topology with input-partition assignments and its state stores. Tasks can run on different threads or instances; a failed task’s work can be reassigned. Each event traverses the processing graph before the next is consumed. section 8.3.4

Topology behavior can be tested without a running Kafka cluster using TopologyTestDriver: supply records through test input topics and check output topics. Tests can also simulate time and inspect intermediate state. section 10.3.1

How Kafka for Architects covers it

How deep it goes

Explains graph nodes, edges, state stores, task assignments, and event traversal through the processing graph.

Examples

Illustrates source-to-sink event flow and custom processing for anomaly detection or fraudulent-transaction detection.

What you can do after

Build it from scratch by defining processors and their connections, or test a topology with TopologyTestDriver.

How it connects

Connects Kafka Streams topologies to Kafka topics, state stores, stream-processing tasks, and the Processor API.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with R, Third EditionNetwork topology is the structural arrangement and connectivity pattern of a neural network's layers and operations.
• 

  liveBookmanning.com
  Secure APIsNetwork topology is the arrangement and connectivity relationships among networks, hosts, and communication components.

Compare how 3 books explain network topology →

Related concepts

• directed acyclic graphkind of · A directed acyclic graph is a graph with directed edges and no directed cycles.
• full mesh topologyincludes · A full mesh topology is a network arrangement in which every node has a direct connection to every other node.
• TensorBoardused by · TensorBoard is a web-based tool for visualizing and monitoring machine-learning experiments and their logged data, including metrics, model graphs, and other run information.
• hypothesis spaceconfigures · A hypothesis space is a set of candidate functions, models, parameter configurations, or rules that a learning or inference procedure can represent or consider.
• star topologyincludes · A star topology is a network arrangement in which multiple peripheral nodes connect and communicate through a central hub or node.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.