12 What’s next for Kafka
This chapter covers
- The evolution of Kafka from a messaging system to a foundational event platform
- Kafka as an orchestration backbone and the rise of low-code and declarative platforms
- Integrating Kafka with emerging runtimes like WebAssembly, serverless, and edge computing
- New architectural directions including diskless Kafka and separation of storage from brokers
- Kafka’s growing role in machine learning workflows and AI systems
12.1 Kafka’s origins: A path to event backbone
In the late 2000s, LinkedIn was facing a data integration crisis as it transitioned from a traditional web application into a distributed platform. The systems behind the platform were generating enormous volumes of data — user profiles, page views, messages — and LinkedIn’s users expected all this information to be fresh and up-to-date with minimal delay.
However, the integration approaches used at the time weren’t designed for this kind of scale or responsiveness. The dominant pattern was batch data movement, which introduced unacceptable delays and complexity. These pipelines weren’t suited for high-volume, low-latency ingestion, and they made it difficult to reuse or reprocess data across different teams and applications.
There was a clear need for a new kind of messaging system — one that would:
- Handle massive volumes of events per second.
- Support low-latency processing for real-time responsiveness.
- Provide durable storage, so consumers could recover data even after downtime.