12 What’s next for Kafka

 

This chapter covers

  • The evolution of Kafka from a messaging system to a foundational event platform
  • Kafka as an orchestration backbone and the rise of low-code and declarative platforms
  • Integrating Kafka with emerging runtimes like WebAssembly, serverless, and edge computing
  • New architectural directions including diskless Kafka and separation of storage from brokers
  • Kafka’s growing role in machine learning workflows and AI systems

12.1 Kafka’s origins: A path to event backbone

In the late 2000s, LinkedIn was facing a data integration crisis as it transitioned from a traditional web application into a distributed platform. The systems behind the platform were generating enormous volumes of data — user profiles, page views, messages — and LinkedIn’s users expected all this information to be fresh and up-to-date with minimal delay.

However, the integration approaches used at the time weren’t designed for this kind of scale or responsiveness. The dominant pattern was batch data movement, which introduced unacceptable delays and complexity. These pipelines weren’t suited for high-volume, low-latency ingestion, and they made it difficult to reuse or reprocess data across different teams and applications.

There was a clear need for a new kind of messaging system — one that would:

  • Handle massive volumes of events per second.
  • Support low-latency processing for real-time responsiveness.
  • Provide durable storage, so consumers could recover data even after downtime.

12.2 Kafka as an orchestration platform

12.3 Integration with new runtimes

12.3.1 Kafka with WebAssembly

12.3.2 Serverless Kafka

12.3.3 Kafka at the edge

12.4 Diskless Kafka: decoupling storage from brokers

12.5 Kafka in AI/ML world

12.5.1 Incremental learning

12.5.2 Feature engineering in motion

12.5.3 Kafka and AI agents

12.6 Summary