A streaming system must keep processing as event volume grows, but adding capacity is not always enough. Kafka handles high rates of small messages across distributed systems; the useful question is where the workload can be parallelized, what must be moved to expand capacity, and whether that complexity serves the business need. section 5.2.7 section 9.2.1
Consumer scalability is bounded by partition count. Within a consumer group, each partition is assigned to only one consumer instance, so adding instances beyond the available partitions cannot increase that group’s partition-level parallelism. Multiple instances in the same group distribute work, making the choice of partition count an architectural decision rather than a detail to leave until deployment. section 4.3.1
Two checks make a scaling decision more concrete:
- Track consumer lag—the gap between produced and committed offsets—to see whether a group is falling behind. An offset-based calculation is approximate when offsets are missing, notably after compaction. section 4.3.1
- If processing needs more concurrency within a consumer, dispatch records to a bounded worker pool sharded by key. This permits parallel work across keys while preserving per-key order, but out-of-order completion makes offset commits harder: only contiguous completed work can be committed safely. section 4.3.1
Broker capacity has a different constraint. Stateless microservices can often add or remove instances with demand; Kafka brokers hold partitions and their data. Adding a broker does not automatically move partitions or balance traffic. Rebalancing transfers data and updates leadership, consuming CPU, disk I/O, network bandwidth, and time; it can also cause latency spikes. Managed services may automate parts of this work without removing the underlying costs. section 9.2.1
Choosing for an online service
A streaming platform generating large volumes of user-interaction and recommendation events is a strong fit for Kafka’s high-throughput processing. A small e-commerce site with few daily orders and no urgent real-time requirement has a less clear case: fault tolerance and future scale may still matter, but so do the expertise and maintenance costs. Kafka is not automatically the best fit for strict ordering, nightly bulk transfers, or random data access. section 5.2.7 section 5.5
Scalability also belongs in deployment planning. On-premises, cloud, and hybrid choices must balance growth against operational complexity, fault tolerance, security, and compliance. Multi-region cloud deployment can ease infrastructure provision, yet introduces replication latency, networking costs, and consistency challenges. Choose capacity and deployment strategy against the required workload and operating model, not an assumption that autoscaling is effortless. section 9.2 section 9.2.1
