chapter two

2 Getting to know Kafka

 

This chapter covers:

  • Sending and receiving your first Kafka message from the command line.
  • Reviewing the high-level architecture of Kafka.
  • Understanding client options and how your applications can communicate to the system.
  • Creating messages with the Java client.

Now that we have a very high-level view of why we would try out Kafka, this is the point at which we want to start and get more concrete with our terms and understanding, and look at the components that make up the whole of this system.

This chapter will also involve us setting up Kafka for the rest of our examples. While a distributed system, Kafka can be installed and run on a single machine. This will give us a starting point to dive into our hands-on use. As is often the case, the real questions start to appear once the hands hit the keyboard.

By the end of this chapter, you will also be able to send and retrieve your first Kafka message from the command line. If you do not have Kafka set up already, please refer to Appendix B to help you get started so you can get Kafka running on your computer. Let’s get started with Kafka and then spend a little more time digging into the details of Kafka’s architecture.

2.1  Kafka’s Hello World: Sending and retrieving our first message

2.2  A quick tour of Kafka

2.2.1  The what and why of ZooKeeper

2.2.2  Kafka’s high-level architecture

2.2.3  The Commit Log

2.3  Various source code packages and what they do

2.3.1  Kafka Stream Package

2.3.2  Connect Package

2.3.3  AdminClient Package

2.3.4  KSQL

2.4  What sort of clients can I use for my own language of choice?

2.5  Terminology of Kafka

2.5.1  What is a streaming process

2.5.2  What exactly once means in our context

2.6  Summary