Introduction

To be successful—in other words, responsive and nimble—modern large-scale systems must efficiently manage, monitor, and react to multiple event streams. Distributed log technologies like Apache Kafka and Amazon Kinesis are now ready for primetime, so there’s never been a better time to restructure your digital business around a centralized “event firehose”. This approach, known as the Unified Log Paradigm (or ULP), aggregates events from disparate source systems, stores them in a unified log, and enables data processing applications to operate on this event stream. The result is a coherent architecture that will unravel the messy bird’s nest of point-to-point connections between the various transactional systems that many businesses currently have in place.

For this quick-start guide, author Alexander Dean has chosen chapters from three Manning books that hone in on the hot topic of analysis of streaming data. The first is “Analytics-on-write” from Event Streams in Action which Dean co-authored with Valentin Crettaz. In this chapter, you’ll see the benefits analytics-on-write has to offer, including queries that are low-latency, can serve many users at once, and are easy to operate.

In “Algorithms for data analysis” from Streaming Data by Andrew G. Psaltis, you’ll zoom up a level and look at querying a stream as well as various summarization algorithms you can use for streaming analytics. You’ll also examine approaches to solving common time-related problems that can arise when analyzing continuously streaming data.

The last chapter, “Streams and state” from Kafka Streams in Action by William P. Bejeck Jr., explores how to extract the most information from Kafka streams through the use of stateful operations—those that recall information they’ve encountered before and connect it to current information. You’ll also dig into joining streams for added insight, the requirements for using state, and the benefits of timestamps when using stateful operations.

This laser-focused guide provides a well-rounded introduction to analyzing streaming data. We hope you find it useful as you consider continuous data processing for your large-scale data-driven applications.