Welcome

Thank you for purchasing Streaming Data Pipelines with Kafka in MEAP.

We are used to getting groceries delivered in under ten minutes and we can access almost any information at any time via search engines, yet most data integrations are implemented with batch pipelines that leave data systems out of sync at most times. Streaming data pipelines are a modern alternative to batch pipelines: They replicate data between data systems like database systems and APIs in real-time and can process them on the way. As a foundational technology of modern, real- time data architectures, streaming data pipelines enable downstream data consumers to always work with fresh data. Unfortunately, you still need to become an expert in many technologies to successfully apply them.

I am writing this book for data engineers, software developers, and others who face use cases for data- integration at their workplace. It aims to provide a complete introduction to the concepts, development, and deployment of streaming data pipelines, with a strong focus on the open-source technology Apache Kafka, the de-facto industry standard for event streaming. The book not only teaches foundational concepts of streaming data pipelines but prepares the reader to successfully implement them at their workplace. Throughout the book, we will help an imaginary e-commerce company to migrate its legacy batch pipelines to a modern streaming architecture and overcome the typical challenges of such migrations.