Cloud elasticity is the ability to add or remove system resources as workload demand changes. In Kafka deployments, cloud-managed services offer more elasticity than on-premises installations, but regulatory requirements may favor private or hybrid architectures. section 9.2.1
Elasticity depends on how Kafka manages infrastructure and persistent state:
- On-premises Kafka provides control over data locality but carries greater infrastructure responsibility and less elasticity. section 9.2.1
- Diskless Kafka separates message serving from remote object storage, allowing brokers to be started or stopped dynamically as traffic changes. section 12.3
- This flexibility comes with early constraints: diskless topics lack log compaction and transactional writes. section 12.3
Scaling with diskless topics
A Kafka deployment can write producer batches to shared object storage instead of broker-local disks. Because brokers no longer maintain persistent local state, capacity can expand or contract with traffic while clients retain the same API. section 12.3
Figure 12.5 Producer path in diskless Kafka
Use the producer-path diagram to trace how records move through a diskless Kafka design. section 12.3
