concept message broker in category system administration

This is an excerpt from Manning's book Securing DevOps. Login to get full access to this book.

A message broker is an application that receives messages from publishers and routes them to consumers. It’s a fancy pipe with some smart logic to decide which consumer gets a copy of a given message.

Figure 7.4 The second layer of the logging pipeline focuses on streaming log events between the collection layer and the analysis layer.

Message brokers are useful for streaming information between logical components and provide a standard interface between layers that may not know about each other. In the collection layer, systems, applications, and various infrastructure components forward their log messages into a handful of known message brokers. The collection layer only needs to know one thing: where to send the logs.

On the other end of the message broker, we have an analysis layer composed of multiple programs that read log events and perform tasks on them. Without the message broker, the collection layer would need to know the location and purpose of each analysis worker to send logs to them. Adding or removing an analyzer would require a reconfiguration of all log collectors, a situation that is clearly suboptimal.

Figure 7.5 zooms into the routing of messages inside a message broker. Three publishers are represented on the left side of the message broker, and three consumers are on the right side. Log events flow through the broker from left to right.

Figure 7.5 Message brokers route messages between publishers (on the left) and consumers (on the right). Consumers subscribe to specific event topics, and message brokers use that information to route events correctly, possibly duplicating events to make sure all consumers get a copy. In this example, both the signature and rate analyzers get a copy of the messages sent by the system-logs publisher.

The publishers send their messages to a stream (sometimes called a topic or an exchange) and forget about the message. Publishers don’t need to keep any state about the messages they send once the message broker has taken ownership of them. Different message broker software tools provide different reliability rules on published messages:

Listing 7.10 Pseudocode of a storage consumer

consumer raw-storage:
 initialization:
  connect to message broker
  subscribe to all topics using wildcard pattern
 processing:
  for each message:
   parse message body
   normalize values into event structure
   insert event structure into database
   acknowledge consumption of message to broker

An important aspect of log consumers is their size: they should be small programs that perform a single task. A sophisticated logging pipeline may have dozens of consumers that do different things, and each runs autonomously. The message broker is the tie that binds them to the logging pipeline.