liveBook · Manning

Memory bound

As explained in Apache Kafka in Action

A bottleneck is a component or resource that limits the throughput or capacity of a system.

A bottleneck is the component or resource that limits system throughput or capacity. For Kafka performance, first identify whether the limit is the consumer, Kafka, or an external dependency; adding partitions cannot fix a slow downstream system. section 6.1.2 section 6.1

Sizing partitions from consumer throughput

If a consumer takes about 100 ms to process each message, it handles roughly 10 messages per second. A peak target of 100 messages per second calls for at least 10 consumers and 10 partitions—provided the consumer is the limiting factor. section 6.1.2

Allow some headroom and choose partition counts that divide easily when scaling consumers. More partitions can increase throughput, but also use client RAM and add operational complexity. Diagnose the limiting component before increasing partition count. section 6.1.2

How Apache Kafka in Action covers it

How deep it goes
Works through consumer-rate sizing and partition trade-offs, including client RAM costs, but not a general memory-bound analysis.
Examples
Uses a 100 ms processing example to estimate consumer capacity and partitions for a 100-message-per-second peak.
What you can do after
Diagnose whether Kafka, consumers, or external dependencies cap throughput, then estimate consumer and partition needs from processing rate.
How it connects
Connects consumer throughput, topic partitioning, consumer groups, external data stores, and brokers.

Also explained in

Compare how 11 books explain memory bound →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Apache Kafka in Action. The section links point to the parts of the book it draws on.