Part 1. A new holistic approach

Today data is streaming all around us, with new data sources coming online daily. If you’re not yet faced with building a real-time data system, it’s only a matter of time before you will be. More and more businesses will depend on being able to process and make decisions on streams of data. This first part of this book looks at a streaming system, from the point of ingestion all the way through delivering the data for display or consumption by other systems.

Chapter 1 begins by introducing streaming data and laying the foundation of terms we will use. Streaming data and real time may mean different things to different people. This chapter clarifies how we will use these terms and defines our architectural blueprint that we will use as our guide throughout the book. At the end of chapter 1, we glance at how the security relates to streaming systems.

The entry point to a streaming system is the collection or ingestion of data. The patterns of collecting data and preventing data loss are our focus throughout chapter 2.

Upon ingestion of data we need to move it as fast as we can to a message queue (or as some may call it, message buffer). The technology used in this tier comes with various levels of durability, delivery semantics, and impact on the producers and consumers of data. Chapter 3 looks at best practices and how to take these features into account.