Scalability is a system’s ability to maintain or improve service as workload or resource capacity grows. In Kafka, partitioning, broker interactions, persistence, replication, and data structures support scalable real-time processing while preserving reliability and performance. section 8.0
Kafka Connect scales data movement by distributing workers and connector tasks:
- Run multiple workers in Distributed Mode with the same group.id; workers coordinate through the rebalancing protocol and redistribute tasks after failures or additions. section 11.3
- Increase capacity by adding workers with the same group.id, allowing connectors to spread independent tasks across the cluster. section 11.3
- Set tasks.max to control parallelism: importing 12 database tables with tasks.max=4 assigns three tables to each task. section 11.3
- Avoid the FileStreamSource connector for production scaling because it reads only local files, which may not exist on other workers. section 11.3
Scaling a global retailer
A large retailer can separate transactional, inventory, and customer-analytics workloads into different Kafka clusters. Each cluster can grow independently without overloading one cluster with diverse workloads. section 18.4.4
Figure 11.4 A Kafka Connect cluster consists of one or multiple workers. Tasks are distributed across workers, and Kafka Connect uses three topics to store its internal state.
Use the cluster diagram to trace how workers coordinate, how tasks are distributed, and how Kafka Connect stores internal state. section 11.3
Figure 2.1 The high-level IT architecture for our online shopping platform
Use the online-shopping architecture to place scalability concerns around orders, payments, product management, and real-time data processing. section 2.1
