concept Kafka in category microservice

This is an excerpt from Manning's book Microservices Security in Action. Login to get full access to this book.

The security model that you develop to protect service-to-service communication should consider the communication channels that cross trust boundaries, as well as how the actual communication takes place between microservices: synchronously or asynchronously. In most cases, synchronous communication happens over HTTP. Asynchronous communication can happen over any kind of messaging system, such as RabbitMQ, Kafka, ActiveMQ, or even Amazon Simple Queue Service (SQS). In chapters 6, 7, and 8, we discuss various security models to secure synchronous communication among microservices, and chapter 9 covers securing event-driven microservices.

In most cases, synchronous communications happen over HTTP. Asynchronous communications can happen over any kind of messaging system such as RabbitMQ, Apache Kafka, NATS, ActiveMQ, or even Amazon SQS. In this chapter, we discuss how to use Kafka and NATS as a message broker, which enables microservices to communicate with each other in an event-driven fashion, and how to secure the communication channels.

Kafka is the most popular messaging system used in many microservice deployments. If you’re interested in learning more about Kafka, we recommend Kafka in Action by Dylan Scott (Manning, 2020). NATS is an open source, lightweight, high-performing messaging system designed for cloud-native applications, which is also an alternative to Kafka. If you’re interested in learning more about NATS, we recommend Practical NATS by Waldemar Quevedo (Apress, 2018).

In this section, we discuss how to develop a microservice in Spring Boot to push events to a Kafka topic. The microservice receives messages via HTTP, and those messages are then converted into events and pushed to a topic on Kafka. You can find all the samples in the https://github.com/microservices-security-in-action/samples GitHub repository. Navigate to the chapter09/sampe01 directory by using your command-line tool and execute the following command from within the sample01 directory to build the Order Processing microservice:

\> mvn clean install

copy

Make sure that you have Kafka running as per the instructions given in section 9.2. Execute the following command to run the Order Processing microservice:

\> mvn spring-boot:run

copy

Once the Order Processing microservice starts, you should see a topic named ORDERS being created on Kafka. You can verify that it’s there by listing the topics in Kafka. Execute the following command in a new terminal window from within the kafka_home location:

Figure 9.6 curl makes an HTTP request to the Order Processing microservice to place an order. After processing its logic, the Order Processing microservice puts an event into the ORDERS topic in Kafka with the details of the order. The consumer process subscribed to the ORDERS topic receives the order’s details through Kafka.

This is an excerpt from Manning's book Enterprise Java Microservices. Login to get full access to this book.

Topics in Kafka relate to a category, or type, of record that can be published and consumed. For instance, you’d use one topic for real-time share prices and a separate topic for measurements from vehicle sensors.

To properly support the model in figure 11.6, you want to be able to convert the database change events into records in Kafka for you to process. Such a solution has the least impact on the Admin microservice while still enabling you to consume its data. What you need is a connector for Kafka that can do this for you.

Figure 11.6. Current Cayambe hybrid monolith

Are any tools available that would make that possible? Why, yes, there are! Debezium is an open source project for streaming changes out of databases into Kafka.

11.3. Deploying and using Kafka for data streaming

Before looking at implementing the microservices to integrate with Kafka, let’s get Kafka up and running on OpenShift! If you don’t have Minishift running already, let’s start it now just as you did in chapter 10:

This is an excerpt from Manning's book Spring Microservices in Action. Login to get full access to this book.

Spring Cloud Stream (https://cloud.spring.io/spring-cloud-stream/) is an enabling technology that allows you to easily integrate lightweight message processing into your microservice. Using Spring Cloud Stream, you can build intelligent microservices that can use asynchronous events as they occur in your application. With Spring Cloud Stream, you can quickly integrate your microservices with message brokers such as RabbitMQ (https://www.rabbitmq.com/) and Kafka (http://kafka.apache.org/).

For this chapter, you’re going to use a lightweight message bus called Kafka (https://kafka.apache.org/). Kafka is a lightweight, highly performant message bus that allows you asynchronously send streams of messages from one application to one or more other applications. Written in Java, Kafka has become the de facto message bus for many cloud-based applications because it’s highly reliable and scalable. Spring Cloud Stream also supports the use of RabbitMQ as a message bus. Both Kafka and RabbitMQ are strong messaging platforms, and I chose Kafka because that’s what I’m most familiar with.

Figure 8.4. When organization service data changes it will publish a message to Kafka.

Listing 8.6. Mapping the licensing service to a message topic in Kafka

copy