Flink

As explained in Kafka for Architects

Apache Flink is an open-source distributed stream-processing framework for stateful computations over bounded and unbounded data streams.

Apache Flink runs data-processing jobs that ingest, transform, and output data. It handles both continuously arriving, unbounded streams and bounded datasets with a defined start and end. This makes it an option when a pipeline needs real-time processing alongside batch-style work. section 8.4.2

Put routing logic between producers and consumers

Suppose downstream services need different kinds of address changes. Kafka brokers do not inspect message content; putting routing in the producer complicates upstream logic, while making every consumer filter wastes effort. A streaming application can read address events, select by address type, and publish to separate shipping and billing topics. Flink is one framework for building this kind of consume-transform-re-emit application. section 1.5.3

A Flink job starts with a source connector and ends with a sink connector. Sources can include Kafka topics, files, and batch database queries; predefined connectors also cover other databases and messaging systems. Compared with Kafka Streams, which centers on Kafka as its source and sink, Flink offers more flexibility for pipelines spanning external systems. That flexibility is a reason to consider it—not a reason to add it to every Kafka pipeline. section 8.4.2 section 8.7

Figure 8.21 The Flink architecture. The Flink Client is responsible for transforming a program into a graph, which is submitted to the JobManager. JobManager converts it into an ExecutionGraph and deploys tasks to available TaskManagers.

The runtime diagram shows how a Flink Client turns a program into a graph, the JobManager prepares its execution graph, and TaskManagers receive the deployed tasks. section 8.4.2

How Kafka for Architects covers it

How deep it goes

Defines jobs, stream bounds, and connector roles, then outlines runtime deployment without a low-level implementation walkthrough.

Examples

Illustrates content-based address routing and a Customer 360 operational data store, including a depicted design without Kafka.

What it assumes

Assumes familiarity with Kafka producers, consumers, and topics while defining Flink jobs and bounded versus unbounded streams.

What you can do after

Evaluate/understand when Flink’s batch-and-stream capabilities and external connectors suit a pipeline better than a Kafka-centered option.

How it connects

Connects Flink to Kafka, Kafka Streams, Kafka Connect, stream processing, batch processing, and exactly-once semantics.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Architecting an Apache Iceberg LakehouseApache Flink is an open-source distributed processing framework for stateful batch and stream computations.

Compare how 2 books explain Flink →

Related concepts

• Apache Kafkauses · Apache Kafka is a distributed event-streaming platform for publishing, storing, processing, and consuming streams of records.
• Kafka Streamsalternative to · Kafka Streams is a client library for building applications that continuously process data from and write data to Apache Kafka topics.
• checkpointinguses · A checkpoint is a designated point in a process at which its state or progress is recorded or evaluated, often to support recovery, continuation, or control.
• streamingused by · Streaming is the incremental handling of a sequence of data items, including their processing, transfer, or consumption as they become available.
• Apache Iceberguses · Apache Iceberg is an open table format for managing large analytical datasets with metadata, schema evolution, snapshots, and transactional operations across distributed or object storage.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.