2 Getting to know Kafka

 

This chapters covers

  • The high-level architecture of Kafka
  • Understanding client options
  • How applications communicate with a broker
  • Producing and consuming your first message
  • Using Kafka clients with a Java application

Now that we have a high-level view of where Kafka shines and why one would use it, let’s dive into the Kafka components that make up the whole system. Apache Kafka is a distributed system at heart, but it is also possible to install and run it on a single host. That gives us a starting point to dive into our sample use cases. As is often the case, the real questions start flowing once the hands hit the keyboard. By the end of this chapter, you will be able to send and retrieve your first Kafka message from the command line. Let’s get started with Kafka and then spend a little more time digging into Kafka’s architectural details.

NOTE

Visit appendix A if you do not have a Kafka cluster to use or are interested in starting one locally on your machine. Appendix A works on updating the default configuration of Apache Kafka and on starting the three brokers we will use in our examples. Confirm that your instances are up and running before attempting any examples in this book! If any examples don’t seem to work, please check the source code on GitHub for tips, errata, and suggestions.

2.1 Producing and consuming a message

2.2 What are brokers?

2.3 Tour of Kafka

2.3.1 Producers and consumers

2.3.2 Topics overview

2.3.3 ZooKeeper usage

2.3.4 Kafka’s high-level architecture

2.3.5 The commit log

2.4 Various source code packages and what they do

2.4.1 Kafka Streams

2.4.2 Kafka Connect

2.4.3 AdminClient package

2.4.4 ksqlDB

2.5 Confluent clients

2.6 Stream processing and terminology

2.6.1 Stream processing

2.6.2 What exactly-once means

Summary

References