Task

As explained in Apache Kafka in Action

A Kafka Streams task is the smallest processing unit, responsible for processing one or more assigned partitions and executing a topology.

A Kafka Streams task is the smallest processing unit in an application. It processes an assigned partition and executes the relevant topology. Because partitions are processed independently, the number of tasks depends on the number of partitions being processed. section 12.6

Partition-based parallelism

Filtering a topic with 12 partitions creates 12 tasks, one per partition. One instance can process all 12; six instances typically process two each. With nine tasks, two instances configured with two and three threads can divide the work across both instances and their threads. section 12.6

Kafka Streams distributes tasks across instances using the Kafka Rebalance Protocol and can distribute them across threads within an instance through num.threads. When an instance fails, other instances receive its tasks and continue the work. This makes task placement central to scaling, failover, and operational capacity planning. section 12.6

Figure 12.13 Kafka Streams applications are divided into tasks that are distributed across different threads on different instances. If an instance fails, another instance takes over the tasks.

Use the task-distribution figure to trace how tasks move across instances and threads, including takeover after an instance failure. section 12.6

How Apache Kafka in Action covers it

How deep it goes

Explains task creation, assignment, threading, rebalancing, and failure takeover, but does not descend into task implementation internals.

Examples

Demonstrates twelve filtering tasks, six-instance distribution, nine-task thread allocation, and failure-driven reassignment.

What it assumes

Assumes Kafka partitions and introduces task parallelism through KStreams, KTables, instances, threads, and rebalancing.

What you can do after

Apply task-based partitioning to reason about Kafka Streams scaling, thread configuration, instance capacity, and failure recovery.

How it connects

Connects tasks with Kafka partitions, KStreams, KTables, Kafka Rebalance Protocol, consumer groups, and Kafka Connect.

liveBookmanning.com

Read Apache Kafka in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Kafka Streams in Action, Second EditionA StreamTask is a Kafka Streams runtime unit that processes records for assigned input partitions and maintains its associated local state.
• 

  liveBookmanning.com
  Kafka for ArchitectsA Kafka Streams task is an independently executable processing unit containing assigned input partitions, state stores, and a graph topology.

Compare how 3 books explain task →

Related concepts

• state storehas part · A state store is a storage component that maintains application or processing state and provides mechanisms to access or update it.
• Kafka Streamspart of · Kafka Streams is a client library for building applications that continuously process data from and write data to Apache Kafka topics.
• connectorproduced by · A connector is a software element or mechanism that links components, applications, or resources to enable or represent their interaction, communication, or data exchange.
• shared-nothing architectureuses · Shared-nothing architecture is a design in which components independently manage resources without sharing mutable state, communicating through explicit interfaces.
• Kafka Connect workerpart of · A Kafka Connect worker is a process that manages connectors and runs their tasks within a Kafka Connect cluster.
• source connectorused by · A source connector is a Kafka Connect connector that reads data from an external source system and publishes records to Kafka.

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.