I/o thread

As explained in Apache Kafka in Action

An I/O thread is a Kafka broker thread that performs broker input and output processing.

An I/O thread is a Kafka broker thread that processes queued broker input and output. Network threads place producer and consumer requests in the request queue; I/O threads then handle the request and place the result in the response queue. section 8.2.1 section 9.2

For a produce request, an I/O thread writes the message batch to the end of the appropriate partition log segment, normally through the operating system’s page cache. Kafka can then wait for in-sync followers before returning the producer response; the I/O thread does not itself guarantee durable disk persistence. section 8.2.1 section 8.5

For a consumer fetch request, an I/O thread reads message batches rather than writing them. The data may come directly from the page cache. If no messages are available, the broker can wait for more data according to fetch.min.bytes and fetch.max.wait.ms before queuing the response. section 9.2

Configure I/O capacity around the broker’s workload:

• Set num.io.threads to match the number of disks when multiple disks are used, supporting throughput across the available storage. section 6.3.1
• Treat tuning as a production concern for brokers handling large data volumes, alongside network-thread, queued-request, and operating-system settings. section 6.3.1
• Do not confuse I/O-thread throughput with disk durability: Kafka normally relies on the operating system to flush page-cache data, while replication supplies reliability. section 8.2.1

Figure 8.3 When a broker receives a produce request (1), it’s put on the request queue (2) and later handled by the I/O threads (3) and written to the page cache (4). The operating system flushes it to disk later. Then, the message stays in purgatory (5) until all in-sync followers acknowledge the message (6). When the wait (7) is over, the response is put on the response queue (8) and sent back to the producer by one of the network threads (9).

Trace a produce request from network reception through the request queue, I/O-thread processing, page-cache storage, follower acknowledgment, and the response queue. section 8.2.1

How Apache Kafka in Action covers it

How deep it goes

Explains request queues, partition files, page-cache access, and replication waits, but defers detailed Kafka data structures.

Examples

Uses produce and fetch requests to show how I/O threads write to or read from partition storage.

What it assumes

Assumes familiarity with Kafka brokers, producers, consumers, network threads, and partition log segments.

What you can do after

Operate/configure broker throughput by relating num.io.threads to disk count and workload.

How it connects

Connects I/O threads with network threads, request queues, response queues, page cache, log segments, replication, and consumer fetch settings.

liveBookmanning.com

Read Apache Kafka in Action on liveBook Get the book at manning.com

Related concepts

• file systemuses · A file system is a software-managed structure and interface for organizing, naming, storing, and accessing files and directories on storage.
• log segmentproduces · A log segment is a file-backed portion of a Kafka partition log containing a contiguous range of records and associated indexes.
• throughputuses · The amount of work, data, or output a system processes or completes per unit of time.
• event loop threadimplements · An event loop thread is a thread that runs an event loop to process asynchronous application events and callbacks.

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.