Chapter 2. The unified log

The previous chapter introduced the idea of events and continuous streams of events and showed that many familiar software platforms and tools have event-oriented underpinnings. We recapped the history of business intelligence and data analytics, before introducing an event-centric data processing architecture, built around something called a unified log. We started to show the why of the unified log with some use cases but stopped short of explaining what a unified log is.

In this chapter, we will start to get hands-on with unified log technology. We will take a simple Java application and show how to update it to send events to a unified log. Understanding the theory and design of unified logs is important too, so we’ll introduce the core attributes of the unified log first.

We have a few unified log implementations to choose from. We’ll pick Apache Kafka, an open source, self-hosted unified log to get us started. With the scene set, we will code up our simple Java application, start configuring Kafka, and then code the integration between our app and Kafka. This process has a few discrete steps:

2.1. Understanding the anatomy of a unified log

All this talk of a unified log, but what exactly is it? A unified log is an append-only, ordered, distributed log that allows a company to centralize its continuous event streams. That’s a lot of jargon in one sentence. Let’s unpack the salient points in the following subsections.

2.1.1. Unified

What does it mean that our log is unified? It means that we have a single deployment of this technology in our company (or division or whatever), with multiple applications sending events to it and reading events from it. The Apache Kafka project (Kafka is a unified log) explains it as follows on its homepage (https://kafka.apache.org/):

Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization.

Having a single unified log does not mean that all events have to be sent to the same event stream—far from it: our unified log can contain many distinct continuous streams of events. It is up to us to define exactly how we map our business processes and applications onto continuous event streams within our unified log, as we will explore further in chapter 3.

Let’s imagine a metropolitan taxi firm that is embracing the unified log wholeheartedly. Several interesting “actors” are involved in this taxi business:

Figure 2.1 demonstrates one possible way of architecting this taxi firm around its new unified log implementation. The three streams share the same unified log; there is no reason for them not to. Applications such as the Dispatch Engine can read from two continuous streams of events and write into another stream.

2.1.2. Append-only

Append-only means that new events are appended to the front of the unified log, but existing events are never updated in place after they’re appended. What about deletion? Events are automatically deleted from the unified log when they age beyond a configured time window, but they cannot be deleted in an ad hoc fashion. This is illustrated in figure 2.2.

Being append-only means that it is much easier for your applications to reason about their interactions with the unified log. If your application has read events up to and including event number 10, you know that you will never have to go back and look at events 1 through 10 again.

Of course, being append-only brings its own challenges: if you make a mistake when generating your events, you cannot simply go into the unified log and apply changes to fix those events, as you might in a conventional relational or NoSQL database. But we can compensate for this limitation by carefully modeling our events, building on our understanding of events from chapter 1; we’ll look at this in much more detail in chapter 5.

2.1.3. Distributed

Distributed might sound a little confusing: is the log unified, or is it distributed? Actually, it’s both! Distributed and unified are referring to different properties of the log. The log is unified because a single unified log implementation is at the heart of the business, as explained previously in section 2.1.1. The unified log is distributed because it lives across a cluster of individual machines.

Clustered software tends to be more complex to set up, run, and reason about than software that lives inside one machine. Why do we distribute the log across a cluster? For two main reasons:

To make it easier to work across a cluster of machines, unified logs tend to divide the events in a given event stream into multiple shards (sometimes referred to as partitions); each shard will be replicated to multiple machines for durability, but one machine will be elected leader to handle all reads and writes. Figure 2.3 depicts this process.

2.1.4. Ordered

Ordered means that the unified log gives each event in a shard a sequential ID number (sometimes called the offset) that uniquely identifies each message within the shard. Keeping the ordering restricted to the shard keeps things much simpler—because there is no need to maintain and share a global ordering across the whole cluster. Figure 2.4 shows an example of the ordering within a three-shard stream.

This ordering gives the unified log much of its power: different applications can each maintain their own cursor position for each shard, telling them which events they have already processed, and thus which events they should process next.

If a unified log did not order events, each consuming application would have to do one of the following:

Both of these alternatives would be extremely painful. In the first case, the number of event IDs to keep track of would become hugely unwieldy, reminiscent of the Jorge Luis Borges short story:

In that Empire, the Art of Cartography attained such Perfection that the map of a single Province occupied the entirety of a City, and the map of the Empire, the entirety of a Province.

Jorge Luis Borges, “Del Rigor en la Ciencia” (1946)

The second option would not be much better. We would lose the immutability of our unified log and make it hard for multiple applications to share the stream, or for the sample application to “replay” events it had already processed. And in both situations, the unified log would have to support random access to individual events from consuming applications. So, ordering the log makes a huge amount of sense.

To recap: you have now seen why the unified log is unified, why it is append-only, why it is ordered, and why, indeed, it is distributed. Hopefully, this has started to clarify how a unified log is architected, so let’s get started using one.

2.2. Introducing our application

With the basic theory of the unified log set out, let’s begin to put the theory to work! In each part of this book, we will be working with a fictitious company that wants to implement a unified log across its business. To keep things interesting, we will choose a company in a different sector each time. Let’s start with a sector that almost all of us have experienced, at least as customers: e-commerce.

Imagine that we work for a sells-everything e-commerce website; let’s call it Nile. The management team at Nile wants the company to become much more dynamic and responsive: Nile’s analysts should have access to up-to-the-minute sales data, and Nile’s systems should react to customer behavior in a timely fashion. As you will see in this part of the book, we can meet their requirements by implementing a unified log.

2.2.1. Identifying our key events

Online shoppers browse products on the Nile website, sometimes adding products to their shopping cart, and sometimes then going on to buy those products through the online checkout. Visitors can do plenty of other things on the website, but Nile’s executives and analysts care most about this Viewing through Buying workflow. Figure 2.5 shows a typical shopper (albeit with somewhat eclectic shopping habits) going through this workflow.

Even though our standard definition of an event as subject-verb-object will be described in section 2.1.7, we can already identify three discrete events in this Viewing through Buying workflow:

To avoid complicating our lives later, this part of the book keeps this workflow deliberately simple, steering clear of more-complex interactions, such as the shopper adjusting the quantity of an item in the shopping basket, or removing an item from the basket at checkout. But no matter: the preceding three events represent the essence of the shopping experience at Nile.

2.2.2. Unified log, e-commerce style

Nile wants to introduce Apache Kafka (https://kafka.apache.org) to implement a unified log across the business. Future chapters cover Kafka in much more detail. For now, it’s important to understand only that Kafka is an open source unified log technology that runs on the Java virtual machine (JVM).

We can define an initial event stream (aka Kafka topic) to record the events generated by shoppers. Let’s call this stream raw-events. In figure 2.6, you can see our three types of events feeding into the stream. To make this a little more realistic, we’ve made a distinction based on how the events are captured:

What should our three event types look like? Unified logs don’t typically have an opinion on the schema for the events passing through them. Instead they treat each event as a “sealed envelope,” a simple array of bytes to be appended to the log and shared as is with applications when requested. It is up to us to define the internal format of our events—a process we call modeling our events.

The rest of this chapter focuses on just a single Nile event type: Shopper views product at time. We will come back to the other two Nile event types in the next two chapters.

2.2.3. Modeling our first event

How should we model our first event, Shopper views product at time? The secret is to realize that our event already has an inherent structure: it follows the grammatical rules of a sentence in the English language. For those of us who are a little rusty on English language grammar, figure 2.7 maps out the key grammatical components of this event, expressed as a sentence.

Let’s go through each of these components in turn:

We now have a way of breaking our event into its component parts, but so far this description is only human-readable. A computer couldn’t easily parse it. We need a way of formalizing this structure further, ideally into a data serialization format that is understandable by humans but also can be parsed by computers.

We have lots of options for serializing data. For this chapter, we will pick JavaScript Object Notation (JSON). JSON has the attractive property of being easily written and read by both people and machines. Many, if not most, developers setting out to model their company’s continuous event streams will start with JSON.

The following listing shows a possible representation for our Shopper views product at time event in JSON.

Our representation of this event in JSON has four properties:

To look at it another way, our event consists of two pieces of event metadata (namely, the event and the timestamp), and two business entities (the shopper and the product). Now that you understand the specific format of our event in JSON, we need somewhere to send them!

2.3. Setting up our unified log

We are going to send the event stream generated by Nile into a unified log. For this, we’re going to pick Apache Kafka. Future chapters cover Kafka in much more detail. For now, it’s just important to understand that Kafka is an open source (Apache License 2.0) unified log technology that runs on the JVM.

Be aware that we are going to start up and leave running multiple pieces of software in the next subsection. Get ready with a few terminal windows (or a tabbed terminal client if you’re lucky).

2.3.1. Downloading and installing Apache Kafka

A Kafka cluster is a powerful piece of technology. But, fortunately, it’s simple to set up a cheap-and-cheerful single-node cluster just for testing purposes. First, download Apache Kafka version 2.0.0:

You will have to access that link in a browser. You cannot use wget or curl to download the file directly. When you have it, un-tar it:

Kafka uses Apache ZooKeeper (https://zookeeper.apache.org) for cluster coordination, among other things. Deploying a production-ready ZooKeeper cluster requires care and attention, but fortunately Kafka comes with a helper script to set up a single-node ZooKeeper instance. Run the script like so:

Now we are ready to start Kafka in a second terminal:

Great, we now have both ZooKeeper and Kafka running. Our bosses at Nile will be pleased.

2.3.2. Creating our stream

Kafka doesn’t use our exact language of continuous event streams. Instead, Kafka producers and consumers interact with topics; you might remember the language of topics from our publish/subscribe example with the NSQ message queue in chapter 1.

Let’s create a new topic in Kafka called raw-events:

Let’s briefly go through the second line of arguments:

We can see our new topic if we run the list command:

If you don’t see raw-events listed or get some kind of Connection Refused error, go back to section 2.3.1 and run through the setup steps in the exact same order again.

2.3.3. Sending and receiving events

Now we are ready to send our first events into the raw-events topic in Nile’s unified log in Kafka. We can do this at the command line with a simple producer script. Let’s start it running:

Per the command-line arguments, we will be sending, or producing, events to the Kafka topic raw-events, which is available from the Kafka broker available on our local server on port 9092. To send in our events, you type them in and press Enter after each one:

Press Ctrl-D to exit. We have sent in three Shopper views product events to our Kafka topic. Now let’s read the same events out of our unified log, using the Kafka command-line consumer script:

Success! Let’s go through these arguments briefly:

This time, press Ctrl-C to exit. As a final test, let’s pretend that Nile has a second application that also wants to consume from the raw-events stream. It’s a key property of a unified log technology such as Kafka that we can have multiple applications reading from the same event stream at their own pace. Let’s simulate this with another call of the consumer script:

Fantastic—our second request to read the raw-events topic from the beginning has returned the exact same three events. This helps illustrate the fact that Kafka is serving as a kind of event stream database. Compare this to a pub/sub message queue, where a single subscriber reading messages “pops them off the queue,” and they are gone for good.

We can now send a well-structured event stream to Apache Kafka—a simple event stream for Nile’s data engineers, analysts, and scientists to work with. In the next chapter, we will start to make this raw event stream even more useful for our coworkers at Nile by performing simple transformations on the events in-stream.

Summary