liveBook · Manning

JobGraph

As explained in Kafka for Architects

A JobGraph is Flink's execution graph representation that defines task dependencies and execution optimizations for a submitted job.

A JobGraph is Flink’s representation of a streaming job after the Flink Client converts the application’s logical graph. It defines task dependencies and includes execution optimizations. The client submits it to the JobManager, which converts it into an ExecutionGraph for distributed execution across TaskManagers. section 8.4.2

From operations to tasks

An application with sources, transformations, and sinks first becomes a logical graph. The client turns that graph into a JobGraph; the JobManager then determines task parallelism, dependencies, and resource requirements as it builds the ExecutionGraph. section 8.4.2

Figure 8.21 The Flink architecture. The Flink Client is responsible for transforming a program into a graph, which is submitted to the JobManager. JobManager converts it into an ExecutionGraph and deploys tasks to available TaskManagers.
A diagram of a task manager

Description automatically generated

The architecture diagram shows where the JobGraph sits between the Flink Client and the JobManager’s ExecutionGraph. section 8.4.2

How Kafka for Architects covers it

How deep it goes
Defines the JobGraph and traces its conversion to an ExecutionGraph, without detailing optimization algorithms.
Examples
Illustrates conversion of source, transformation, and sink operations through logical and execution graphs.
What it assumes
Assumes a Flink streaming application and introduces its logical-graph-to-ExecutionGraph path.
What you can do after
Evaluate/understand where task dependencies are represented before a Flink job is planned for distributed execution.
How it connects
Connects the logical graph, Flink Client, JobManager, ExecutionGraph, and TaskManagers.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.