about this book
We wrote Apache Kafka in Action to share our hands-on experience and make your journey with Kafka both effective and enjoyable. Through our years of training professionals and implementing Kafka in organizations, we’ve learned what works and what doesn’t. While we focus on building practical knowledge—from your first steps with Kafka through to running production systems—we also dive deep into the theoretical foundations that are crucial for success with Kafka. Think of this book as the guide we wish we had when we started our own Kafka journey, combining essential theory with real-world expertise. Throughout the book, you’ll find illustrative diagrams, practical tips, and easy-to-follow code examples that you can quickly implement yourself.
Who should read this book
We wrote this book for IT professionals who want to grow their Kafka expertise, whether they’re just starting out or already working with Kafka systems. While a general understanding of modern IT architectures and distributed systems is helpful, you don’t need any prior Kafka experience. Our goal is to meet you where you are and build your knowledge step by step. Whether you’re a developer, system administrator, architect, or technical team lead looking to enhance your data infrastructure, you’ll find practical guidance throughout the book. Even if you’re a seasoned Kafka expert, you’ll likely discover new insights and “Aha!” moments as we explore advanced patterns, organizational challenges, and lesser-known features.
How this book is organized: A road map
Apache Kafka in Action is organized into four parts comprising 18 chapters, each building upon the previous to create a comprehensive understanding of Kafka from basics to enterprise implementation:
- Part 1: Getting started (chapters 1–2) introduces you to the world of Apache Kafka. We begin by explaining what Kafka is, its architecture, and its role in modern data architectures. Through practical examples using a running Kafka cluster, you’ll learn how to work with basic Kafka operations, gaining hands-on experience with topics, producers, and consumers.
- Part 2: Concepts (chapters 3–6) delves into the essential building blocks of Kafka. We explore the details of topics and messages, examine Kafka’s role as a distributed log, and investigate how Kafka achieves reliability through replication and transactions. The section concludes with a deep dive into performance optimization, covering crucial aspects such as partitioning strategies and configuration tuning.
- Part 3: Kafka deep dive (chapters 7–10) takes you behind the scenes of Kafka’s operations. You’ll learn the intricacies of cluster management, discover how messages are produced and persisted, understand the mechanics of message consumption, and master message cleanup strategies. This section provides the technical depth needed to troubleshoot problems and optimize your Kafka deployment.
- Part 4: Kafka in enterprise use (chapters 11–18) bridges the gap between theory and real-world implementation. We cover essential enterprise topics such as system integration with Kafka Connect, stream processing, governance, and reference architectures. You’ll learn practical skills for monitoring, performing disaster recovery, and choosing between different technologies. The section concludes with guidance on avoiding common pitfalls and implementing Kafka successfully in modern architectures.
While the book is designed to be read sequentially, experienced practitioners may choose different paths. If you’re already familiar with Kafka basics, you might skim part 1, but we recommend reviewing part 2 as it contains insights that even experienced users often find valuable. Technical leaders focusing on architectural decisions might prefer to concentrate on parts 1, 2, and 4, using part 3’s technical deep-dive as a reference when needed. Throughout the book, you’ll find numerous practical examples that demonstrate real-world applications of the concepts being discussed.
For those new to Kafka or needing guidance on setup, appendix A provides detailed instructions for creating a test environment, while appendix B covers monitoring setup procedures. These appendixes serve as practical references you can return to whenever needed.
About the code
This book contains many examples of source code listings in line with normal text. Source code is formatted in a fixed-width font like this to separate it from ordinary text.
In many cases, the original source code has been reformatted; we’ve added line breaks and reworked indentation to accommodate the available page space in the book. In rare cases, even this wasn’t enough, and listings include line-continuation markers (➥). Code annotations accompany many of the listings, highlighting important concepts. You can get executable snippets of code from the liveBook (online) version of this book at https://livebook.manning.com/book/apache-kafka-in-action.
liveBook discussion forum
Purchase of Apache Kafka in Action includes free access to liveBook, Manning’s online reading platform. Using liveBook’s exclusive discussion features, you can attach comments to the book globally or to specific sections or paragraphs. It’s a snap to make notes for yourself, ask and answer technical questions, and receive help from the author and other users. To access the forum, go to https://livebook.manning.com/book/apache-kafka-in-action/discussion. You can also learn more about Manning’s forums and the rules of conduct at https://livebook.manning.com/discussion.
Manning’s commitment to our readers is to provide a venue where a meaningful dialogue between individual readers and between readers and the authors can take place. It’s not a commitment to any specific amount of participation on the part of the authors, whose contribution to the forum remains voluntary (and unpaid). We suggest you try asking the authors some challenging questions lest their interest stray! The forum and the archives of previous discussions will be accessible from the publisher’s website as long as the book is in print.
In this book
appendix A Setting up a Kafka test environment appendix B Monitoring setup