In Kafka Streams, a task is the single-threaded runtime unit that processes records for its assigned work and, when stateful, owns its local state store. Tasks follow a shared-nothing model: other tasks and stream threads do not share their data or stores, so the owning task alone accesses its state without locking. section 7.4.3 section 10.2.4
Task count is driven by input-partition count. More partitions can produce more tasks and allow more stream threads to process records, but extra tasks improve throughput only when keys are distributed reasonably evenly across partitions. section 7.2.5
Use this model when planning stateful processing and capacity:
- Treat each task as an isolated owner of its state store; state access is not concurrent between tasks. section 7.4.3
- Scale processing by increasing usable partition parallelism, then run sufficient application instances or threads to receive task assignments. section 7.2.5
- Check key skew before repartitioning: a dominant key range can leave one task carrying most of the work. section 7.2.5
Repartition for higher task parallelism
A stream can call repartition with 10 partitions to create an application with 10 tasks and up to 10 record-processing stream threads. The value is illustrative rather than a production recommendation; choose partitioning with the expected key distribution in mind. section 7.2.5
A Kafka Connect task has a related but distinct role. A connector generates SourceTask or SinkTask configurations, and Connect starts and distributes task instances across workers. Each instance runs in its own thread; tasks.max is a ceiling, not a promise that every allowed task will exist. section 5.3 section 5.6.1
Figure 7.20 Stateful tasks have a state store assigned to them.
Inspect the one-to-one ownership relationship between a stateful task and its assigned state store. section 7.4.3
