concept tuple tree in category storm

This is an excerpt from Manning's book Storm Applied: Strategies for real-time event processing. Login to get full access to this book.

Storm uses special “acker” tasks to keep track of tuple trees in order to determine whether a spout tuple has been fully processed. If an acker task sees a tuple tree is complete, it’ll send a message to the spout that originally emitted the tuple, resulting in that spout’s ack method being called.

A tuple that’s emitted from a spout can result in many additional tuples being emitted by the downstream bolts. This creates a tuple tree, with the tuple emitted by the spout acting as the root. Storm creates and tracks a tuple tree for every tuple emitted by the spout. Storm will consider a tuple emitted by a spout to be fully processed when all the leaves in the tree for that tuple have been marked as processed. Here are two things you need to do with the Storm API to make sure Storm can create and track the tuple tree:

All of the leaves in a tuple tree aren’t marked as processed (acked) within a certain time frame. This time frame is configurable at the topology level via the TOPOLOGY_MESSAGE_TIMEOUT_SECS setting, which defaults to 30 seconds. Here’s how you’d override this default when building your topology:

Config config = new Config();
config.setMessageTimeoutSecs(60);.

copy

Config config = new Config();
config.setMessageTimeoutSecs(60);.

copy

A tuple is manually failed in a bolt, which triggers an immediate failure of the tuple tree.

We keep mentioning the phrase tuple tree, so let’s walk through the life of a tuple tree in our topology to show you how this works.

Figure 4.5 starts things off by showing the initial state of the tuple tree after our spout emits a tuple. We have a tree with a single root node.

Figure 4.5. Initial state of the tuple tree

The first bolt in the stream is the AuthorizeCreditCard bolt. This bolt will perform the authorization and then emit a new tuple. Figure 4.6 shows the tuple tree after emitting.

Figure 4.6. Tuple tree after the AuthorizeCreditCard bolt emits a tuple

We’ll need to ack the input tuple in the AuthorizeCreditCard bolt so Storm can mark that tuple as processed. Figure 4.7 shows the tuple tree after this ack has been performed.

Figure 4.7. Tuple tree after the AuthorizeCreditCard bolt acks its input tuple

Once a tuple has been emitted by the AuthorizeCreditCard bolt, it makes its way to the ProcessedOrderNotification bolt. This bolt doesn’t emit a tuple, so no tuples will be added to the tuple tree. But we do need to ack the input tuple and thus tell Storm this bolt has completed processing. Figure 4.8 shows the tuple tree after this ack has been performed. At this point the tuple is considered fully processed.

Figure 4.8. Tuple tree after the ProcessedOrderNotification bolt acks its input tuple

With a clear definition of a tuple tree in mind, let’s move on to the code that’s needed in our bolts for anchoring and acking. We’ll also discuss failing tuples and the various types of errors we need to watch out for.