2 How to Speak Rabbit: The AMQ Protocol

The process that RabbitMQ and client libraries go through in order to get a message from your application into RabbitMQ and from RabbitMQ into consumer applications can be complex. If you are processing missing critical information, such as sales data, ensuring the reliability of delivering the canonical source of information about the sale should be a top priority. At the protocol level, the AMQP specification defines the semantics for allowing client and broker to negotiate and speak to each other about the process for relaying your information. Oftentimes the lexicon defined in the AMQP specification bubbles its way up into RabbitMQ client libraries, where the classes and methods used by applications communicating with RabbitMQ mirror the protocol level classes and methods. Understanding how this communication takes place will help you learn not just the “how” of communicating with RabbitMQ but also the “why.”

Even though the commands in client libraries tend to resemble or even directly copy the actions defined in the AMQP specification, most client libraries attempt to hide the complexity of communicating via AMQ Protocol. This tends to be a good thing when you’re looking to write an application and you do not want worry about the intricacies of how things work. However, skipping over the technical foundation of what RabbitMQ clients are doing is not very helpful when you want to truly understand what is going on with your application. Whether you want to know why your application is slower to publish than you might expect or you just want to know what steps a client must take in order to establish that first connection with RabbitMQ, knowing how your client is talking to RabbitMQ will make that process much easier.

To better illustrate the how and why, in this chapter you will learn how AMQP splits communication between the client and broker into chunks of data called frames and how these frames detail the actions your client application wants RabbitMQ to take and the actions RabbitMQ wants your client application to take. In addition, you’ll learn how these frames are constructed at the protocol level, and how when working together they provide the mechanism by which messages are delivered and consumed.

Building on this information, you will write your first application in Python using a RabbitMQ client library written as a teaching aide for this book. This application will use the AMQ protocol to define an exchange and queue and then bind them together. Finally, you will write a consumer application that will read the messages from the newly defined queue and print the contents of the message. If you’re already comfortable doing these things, you should still dive in to this chapter. I found that it was only after I fully understood the semantics of the AMQ protocol, the “why” instead of just the “how,” that I understood RabbitMQ.

2.1   AMQP as an RPC transport

As an AMQP broker, RabbitMQ speaks a strict dialect for communication, utilizing a remote procedure call (RPC) pattern in nearly every aspect of communication with the core product. A remote procedure call is a type of communication between computers that allows one computer to execute a program or its methods on the other. If you’ve done web programming where you’re talking to a remote API, you’re using a common RPC pattern.

However, the RPC conversations that take place when communicating with RabbitMQ are unlike most web-based API calls. In most web API definitions, there are RPC conversations where the client issues commands and the server responds. The server does not issue commands back to the client. In the AMQP specification, both the server and the client can issue commands. For a client application, this means that it should be listening for communication from the server that may have little to do with what the client application is doing. To illustrate how RPC works when a client is talking to RabbitMQ, consider the connection negotiation process.

2.1.1   Kicking off the conversation

When you are communicating with someone new in a foreign country for the first time, it is inevitable that one of you will kick off the conversation with a greeting. Something that lets both you and the other person know if you’re both capable of speaking the same language. When speaking AMQP, this greeting is called the Protocol Header and is sent by the client to the server. This greeting should not be considered a request however, as unlike the rest of the conversation that will take place, it is not a command. It is RabbitMQ that starts the command/response sequence by replying to the greeting with a Connection.Start command and the client responds to the RPC request with Connection.StartOk (figure 2.1) response frame.

The full conversation to initiate a connection is not terribly important unless you are writing a client library, but it is worth noting that to fully connect to RabbitMQ, there is a sequence of three synchronous RPC requests to start, tune, and open the connection. Once this sequence has finished, RabbitMQ will be ready for your application to make requests. There are a whole range of different commands your application can send to RabbitMQ and RabbitMQ can send to your client. You’ll learn a small subset of these commands later in the chapter, but before that happens, you have to open a channel.

2.1.2   Tuning in to the right channel

Similar in concept to channels on a two-way radio, the AMQP specification defines channels for communicating with RabbitMQ. Two-way radios transmit information to each other using the airwaves as the connection between them. In AMQP, channels use the negotiated AMQP connection as the conduit for transmitting information to each other; and like channels on a two-way radio, they provide isolation from other conversations that are happening. A single AMQP connection can have multiple channels allowing for multiple conversations between a client and server to take place. In technical terms, this is called multiplexing and can be useful for multi-threaded or asynchronous applications that perform multiple tasks.

2.2   AMQP’s RPC Frame Structure

Very similar in concept to object-oriented programming in languages such as C++, Java, and Python, the AMQ protocol uses classes and methods, referred to as AMQP Commands, to create a common language between clients and servers. The classes in AMQP define a scope of functionality, and each class contains methods that perform different tasks. In the connection negotiation process, the RabbitMQ server sends a Connection.Start command, marshaled into a frame, to the client. As is illustrated in Figure 2.2, the Connection.Start command is comprised of two components, the AMQP class and method.

While there are many commands in the AMQP specification, if you’re like me, you want to skip through all of that and get to the important bits of sending and receiving messages. It’s important, however to understand how the commands you will be sending and receiving with RabbitMQ are represented on the wire to truly appreciate what is happening in your applications.

2.2.1   AMQP frame components

In sending commands to and from RabbitMQ all of the arguments required to execute them are encapsulated data structures called frames that encode the data for transmission. Frames provide an efficient way for the command and its arguments to be encoded and delimited on the wire. You can think of frames like freight cars on a train. As a generalization, freight cars have the same basic structure and are differentiated by what they contain. The same is true with low-level AMQP frames. As Figure 2.3 illustrates, a low-level AMQP frame is comprised of five distinct components:

A low level AMQP frame starts off with three fields, referred to as a frame header when combined together. The first field is a single byte indicating the frame type while the second field specifies the channel the frame is for. The third field carries the byte-size of the frame payload. The frame header, along with the end-byte marker creates the structure for the frame. Carried inside the frame, after the header and before the end-byte marker is the frame payload. Much like the freight car protecting its contents on a train, the frame is designed to protect the integrity of the content it carries.

2.2.2   Types of frames

The AMQP specification defines five types of frames: a protocol header frame, a method frame, a content header frame, a body frame, and a heartbeat frame. While each frame type has a distinct purpose, some are used much more frequently than others.

While the protocol header and heartbeat frames are generally abstracted away from developers when using a client libraries, the method, content header and body frames and their constructs are usually surfaced when writing applications that communicate with RabbitMQ. In the next section, you will learn how messages that are sent into and received by RabbitMQ are marshaled into a method frame, a content header frame and one or more body frames.

2.2.3   Marshaling messages into frames

When publishing a message to RabbitMQ, the method, header, and body frames are used. The first frame sent is the method frame carrying the command and the parameters required to execute it, such as the exchange and routing key. Following the method frame are the content frames: a content header and body. The content header frame contains the message properties along with the body size. The AMQ protocol has a maximum frame size, and if the body of your message sending exceeds that size, the content will be split into multiple body frames. These frames are always sent in the same order over the wire: a method frame, content header frame, and one or more body frames (figure 2.4).

As figure 2.4 illustrates, when sending a message to RabbitMQ, a Basic.Publish command is sent in the method frame and is followed by a content header frame with the message’s properties, such as the message’s content-type and the time the message was sent at. These properties are encapsulated in a data structure defined in the AMQP specification as Basic.Properties. Finally the content of your message is marshaled into the appropriate number of body frames.

In order to be more efficient and minimize the size of the data being transferred, the content in the method frame and content header frame is binary packed data and is not human readable. Unlike the method and header frames, the message content carried inside the body frame is not packed or encoded in any way and may be anything from plain-text to binary image data. To further illustrate the anatomy of an AMQP message, let’s examine these three frame types in more detail.

2.2.4   The anatomy of a method frame

Method frames carry with them the class and method your RPC request is going to make as well as the arguments that are being passed along for processing. In figure 2.5, the method frame carrying a Basic.Publish command carries the binary packed data describing the command and the request arguments that are passing along with it. The first two fields are numeric representations of the Basic class and the Publish method. These fields are followed by the string values for the exchange name and the routing key. As previously mentioned, these attributes instruct RabbitMQ on how to route a message. The mandatory flag tells RabbitMQ that the message must be delivered or the publishing of the message should fail.

Each data value in the method frame payload is encoded in a data-type specific format. This format is designed to minimize byte size on the wire, ensure data integrity and ensure that data marshaling and unmarshaling are as fast as possible. The actual format varies depending on the data type, but it is usually a single byte followed by numeric data or a single byte followed by a byte size field and then text data.

2.2.5   The content header frame

The headers that are sent along after the method frame carry more than the data that tells RabbitMQ how big your message is. As illustrated in figure 2.6, the header frame also carries attributes about your message that are used to describe the message to both the RabbitMQ server and to any application that may receive it. These attributes as values in a Basic.Properties table may contain data that describes the content of your message or they can be completely blank. Most client libraries will pre-populate a minimal set of fields such as the content-type and the delivery mode.

Properties are a powerful tool in composing your message. It can be used to create a contract between publishers and consumers about the content in the message, allowing for a large amount of specificity about the message. You will learn about Basic.Properties and the various possible uses for each fields the data structure can carry in chapter 3.

2.2.6   The body frame

The body frame for a message is agnostic to the type of data being transferred and may contain either binary or text data. Whether you are sending binary data such as a JPEG image or you are sending serialized data in a JSON or XML format, the message body frame is the structure in the message that carries the actual message data (figure 2.7).

Together the message properties and body form a powerful encapsulation format for your data. Marrying the descriptive attributes of the message with the content agnostic body ensures you may use RabbitMQ for any type of data that you deem appropriate.

2.3   Putting the protocol to use

There are a few configuration related steps that must occur prior to your being able to publish messages into a queue. At a minimum you must set up both an exchange and a queue and then bind them together. But before you actually perform those steps, let’s review what needs to happen at a protocol level to enable a message to be published, routed, queued, and delivered, starting with setting up an exchange for routing messages.

2.3.1   Declaring an exchange

Exchanges, like queues, are first-rate citizens in the AMQ model. As such, each has its own class in the AMQP specification. Exchanges are created using the Exchange.Declare command. Exchange.Declare has multiple arguments that define the name of the exchange, its type and other metadata that may be used for message processing.

Once the command has been sent and RabbitMQ has created the exchange, an Exchange.DeclareOk method frame is sent in response (figure 2.8). If for whatever reason the command should fail, it’s important to note that RabbitMQ will close the channel the Exchange.Declare command was sent on by sending a Channel.Close command. This response will include a numeric reply code and text value indicating why the Exchange.Declare failed and the channel was closed.

2.3.2   Declaring a queue

Once the exchange has been created, it’s time to create a queue. To create a queue, a Queue.Declare command is sent to RabbitMQ. Like the Exchange.Declare command, there is a simple communication sequence that takes place (figure 2.9) and also like the Exchange.Declare command, should the Queue.Declare command fail, the channel will be closed.

When declaring a queue, there is no harm in issuing the same Queue.Declare command more than once. RabbitMQ will consider subsequent queue declares to be passive, and returns useful information about the queue, such as the number of pending messages in the queue and the number of consumers subscribed to it.

2.3.3   Binding a queue to an exchange

Once the exchange and queue have been created, it’s time to bind them together. Like with Queue.Declare, the command to bind a queue to an exchange, Queue.Bind can only specify one queue at a time. Similar in sequence to the Exchange.Declare and Queue.Declare commands, after issuing a Queue.Bind command your application will receive a Queue.BindOk method frame if it was processed successfully (figure 2.10).

As basic examples of the RPC interactions between a RabbitMQ server and a client, the Exchange.Declare, Queue.Declare and Queue.Bind commands describe a common pattern that that is mimicked by all synchronous commands in the AMQP specification. However, there are a few asynchronous commands that break from the simple “Action” and “ActionOk” pattern. These commands deal with sending and receiving messages from RabbitMQ.

2.3.4   Publishing a message to RabbitMQ

As you previously learned, when publishing messages to RabbitMQ, there are multiple frames encapsulating the message data that are sent to the server. Before the actual message content ever reaches RabbitMQ, the client application is sending a Basic.Publish method frame, a content header frame and at least one body frame (figure 2.11).

When RabbitMQ receives all of the frames for a message, it will inspect the information it needs from the method frame before determining the next steps. The Basic.Pubish method frame carries with it the exchange name and routing key for the message. When evaluating this data, RabbitMQ will try and match the exchange name in the Basic.Publish frame against its database of configured exchanges.

When RabbitMQ finds a match to the exchange name in the Basic.Properties method frame, it evaluates the bindings in the exchange, looking to match queues with the routing key. When the criterion for a message matches to any bound queues, the RabbitMQ server will enqueue the message to the proper queues in a first-in-first-out (FIFO) order. Instead of putting the actual message itself into a queue data structure, a reference to the message is added to the queue. When RabbitMQ is ready to deliver the message, it will use the reference to compose the marshaled message and send it over the wire. This provides a substantial optimization for messages that are published to multiple queues. By only holding one instance of the message, it takes less physical memory when published to multiple destinations. The disposition of a message in a queue, whether consumed, expired or sitting idle will not impact the disposition of that message in any other queue. Once RabbitMQ no longer needs the message because all copies of it have been delivered or removed, the single copy of the message data will be removed from memory in RabbitMQ.

By default, as long as there are no consumers listing to the queue, messages will be stored in the queue. As you add more messages, the queue will grow in size. RabbitMQ can keep these messages in memory or write them to disk, depending the delivery-mode property specified in the message’s Basic.Properties. The delivery-mode property is so important that it will not only be discussed in more detail in the next chapter but it will also be covered with even more detail in chapter 4.

2.3.5   Consuming messages from RabbitMQ

Once a published message has been routed and enqueued to one or more queues, there are not many details left to discuss about it but its consumption. In order to consume messages from a queue in RabbitMQ, a consumer application subscribes to a queue in RabbitMQ by issuing a Basic.Consume command. Like the other synchronous commands, the server will respond with Basic.ConsumeOk to let the client know it’s going open the floodgates and release a torrent of messages, or at least a trickle. At RabbitMQ’s discretion, the consumer start receiving messages in the unsurprising form of Basic.Deliver methods and their content header and body frame counter parts (figure 2.12).

Once the Basic.Consume has been issued, it will stay active until one of a few things occurs. If a consumer would like to stop receiving messages, it can issue a Basic.Cancel command. Sending a Basic.Cancel command tells RabbitMQ that the consumer would like to stop receiving messages. It’s worth noting that when sending this command, which issued asynchronously while RabbitMQ may still be sending messages, a consumer can receive any number of the messages that RabbitMQ has pre-allocated for it prior to receiving a Basic.CancelOk response frame.

When consuming messages, there are several settings that let RabbitMQ know how you want to receive them. One such setting is the no_ack argument for the Basic.Consume command. When set to true, RabbitMQ will send messages continuously until the consumer sends a Basic.Cancel command or the consumer is disconnected. However, if the no_ack flag is set to false, a consumer must acknowledge each message that it receives. It does so by sending a Basic.Ack RPC request (figure 2.13).

When the Basic.Ack response frame is sent, the consumer must pass with it an argument from the Basic.Deliver method frame called the delivery tag. RabbitMQ uses the delivery tag along with the channel as a unique identifier. This identifier is used to communicate message acknowledgement, rejection and negative acknowledgement. You’ll learn more about these options in chapter 5.

2.4   Writing a message publisher in Python

With a healthy knowledge of AMQP fundamentals under your belt, it’s time to turn theory into practice and write both a publisher and consumer. To do this we will use the rabbitpy library. While there are many libraries for communicating with RabbitMQ, I created rabbitpy as a teaching aid for this book to keep the programming examples simple and concise while attempting to stay true to the AMQP command syntax. If you have not done so yet, please install rabbitpy by following the instructions in Appendix A.

To start our exercise, you’ll make use of the IPython Notebook Server installed as part of the RabbitMQ in Depth virtual machine. If you’ve yet to do so, please follow the steps outlined in Appendix A to setup the virtual machine on your local computer. Open your browser to http://localhost:8888 and you should see a page similar to the following:

The “2.4 Publisher Example” notebook in the index contains all of the code outlined in this page in order to communicate with RabbitMQ we must import the rabbitpy library so that the Python interpreter allows us to use it:

If you press the play button or “Run Cell” button in the toolbar or if you press shift + enter, the cell containing that code will execute. In the first cell of the notebook, the rabbitpy library will be imported. You should also have seen the asterisk (*) change to the number 1. The active cell has automatically advanced from the first one to the next one. As you read through this example code, you should execute each cell as you encounter it, advancing you through the code in the IPython Notebook.

Now, with the library is imported, you will need to create an AMQP connection URL. The format for the URL is very similar to the format used for HTTP requests:

This AMQP URL specifies you will connect over a normal AMQP connection using the username guest and the password guest. It will connect you to localhost on port number 5672 and the default “/” vhost. This URL expects that you will be connecting to RabbitMQ on your local machine with the default configuration. If you have set up RabbitMQ on a remote server or have changed the configuration of the RabbitMQ broker, you will have to change the values accordingly.

Now that the URL has been defined, it is time to open a connection to RabbitMQ:

If you did not receive an exception, you are now connected to RabbitMQ. If you did receive one, than the most likely scenario is that RabbitMQ is not running on your local machine. Please ensure that it is running and then try again. If you are successfully connected, it’s time to open a channel to communicate with RabbitMQ on:

With the channel open, you will now declare an exchange by creating a new instance of the rabbitpy.Exchange class. Pass in the channel and the name of the exchange you would like to create. I’d use chapter2-example for now.

Once constructed, use the exchange object’s declare method to send the command, declaring the exchange in RabbitMQ:

Now that you’ve declared the exchange, you can setup the queue and bind it to the exchange. To do this, first you create the Queue object, passing in the channel and the name of the queue. In the example that follows, the name of the queue is example.

Once the object has been created and the instance returned as the queue variable, you will send the Queue.Declare command to RabbitMQ using the declare method. What you should see is an output line that has a Python tuple data structure with the number of messages in the queue and the number of consumers for the queue. A tuple is an immutable set of python objects. In this case they are integer values.

Now that the queue has been created, you must bind it in order for it to receive messages. To bind the queue to the exchange, send the Queue.Bind command by invoking the queue object’s bind method, passing in the exchange and the routing key. In the example below, the routing key is example-routing-key. When the execution of this cell returns, you should see the output True indicating that the binding was successful.

In your application, I recommend that you use semantically appropriate period-delimited keywords to namespace your routing keys. The Zen of Python states that “Namespaces are one honking great idea – let’s do more of those!” and this is true in RabbitMQ as well. By using period-delimited keywords, you will be able to route messages based upon patterns and sub-sections of the routing-key. You will learn more about this in chapter 6.

With your exchange and queue created and bound, you can now publish test messages into RabbitMQ that will be stored in the example queue. To make sure you have enough messages to play with, the example below publishes 10 test messages into the queue.

To publish test messages, in each loop iteration, a new rabbitpy.Message object is created, passing in the channel, a message body and a dictionary of message properties. Once created, the publish method is invoked creating the Basic.Publish method frame, the content header frame and one body frame, delivering them all to RabbitMQ.

Did you receive any errors? You should now go to the RabbitMQ web management console and see if your messages made it into the queue. Try opening your web browser and visiting the management UI at http://localhost:15672/#/queues/%2F/example.  Like with the code example, if your broker is on a different machine, change localhost in the URL to the appropriate server. Once authenticated, you should see a page resembling the screenshot in figure 2.15.

If you look towards the bottom of the page, you will see the “Get Messages” section. If you change the Messages field value from 1 to 10, and hit “Get Messages” you should see each of the 10 messages you previously published. Make sure you leave the “Requeue” field value set to “Yes.” It tells RabbitMQ to add the messages back into the queue when RabbitMQ is retrieving them to display them in the management UI. If you didn’t, don’t worry, just go back and re-run the publishing code.

2.5   Getting messages from RabbitMQ

With publishing messages under your belt, it is time to retrieve them. To get messages from RabbitMQ, the following listing pulls together the repetitive, yet import connection elements from the publishing code discussed in the last section. This code is in “2.5 Basic.Get Example” notebook. This notebook has 6 cells in when using the IPython Notebook interface. You can click the “Cell” dropdown and then “Run All” to not have to run each cell like in the previous example.

After typing in and executing the consumer code above, you should see each of the 10 messages you previously published. If you were looking closely, you may have noticed that while you did not specify message_id or the timestamp properties when publishing the messages, yet each message printed from the consumer has them. The rabbitpy client library will automatically populate these properties for you if you do not specify them. In addition, had you sent a Python dict in as the message, rabbitpy automatically serializes the data as JSON and sets the content-type property as application/json.

2.6   Summary

The AMQP 0.9.1 specification defines a communication protocol that uses RPC style commands to communicate between the RabbitMQ server and client. By having reviewed how these commands are framed and learning how the protocol functions, you should be better equipped for writing and troubleshooting applications that interact with RabbitMQ. By making it this far, you’ve covered a large majority of the process that goes on when communicating with RabbitMQ for publishing and consuming messages. Many applications contain little more code than what you’ve already implemented to work with your RabbitMQ instance. In the next chapter you will learn even more about using message properties, allowing for your publishers and consumers to use a common contract for the messages your applications exchange.