A network thread is a Kafka broker thread that handles network communication. It receives producer and consumer requests, places them on the request queue, and sends completed responses from the response queue. section 8.2.1 section 9.2
For a produce request, the network thread receives and authorizes the request before queueing it for I/O threads. Those threads write messages to the partition’s current log segment, while a network thread later returns the response after the required wait. section 8.2.1 section 8.5
For a fetch request, the network thread performs the same queueing role. I/O threads read batches from the filesystem or page cache, and a network thread sends the response after fetch conditions are satisfied. section 9.2
Tune network threads according to the broker’s workload:
- Start with the default of three network threads, but consider six when TLS secures data traffic. section 6.3.1
- Treat network threads as one part of request capacity; overloaded I/O threads can still make requests wait in the queue. section 8.2.1
- Use workload measurements before increasing the setting, particularly in production environments handling large data volumes. section 6.3.1
Produce and fetch flows
A producer sends a message to the broker. A network thread queues it, an I/O thread writes it to the page cache, and a network thread eventually returns the acknowledgment. A consumer fetch follows the same entry and exit path, but I/O threads read batches instead of writing them.
Figure 8.3 When a broker receives a produce request (1), it’s put on the request queue (2) and later handled by the I/O threads (3) and written to the page cache (4). The operating system flushes it to disk later. Then, the message stays in purgatory (5) until all in-sync followers acknowledge the message (6). When the wait (7) is over, the response is put on the response queue (8) and sent back to the producer by one of the network threads (9).
Trace the complete produce-request path from network reception through request queuing, page-cache writing, replication waiting, and response delivery. section 8.2.1
Figure 9.2 The broker undergoes a similar process when receiving a fetch request as that of when receiving a produce request. However, instead of writing data to the disk, we need to read from it. Additionally, we don’t need to wait for other brokers, but rather for more messages depending on the fetch.max.wait.ms and fetch.min.bytes settings.
Compare the fetch path with produce handling and see how fetch.min.bytes and fetch.max.wait.ms affect response timing. section 9.2
