chapter eight

8 Stream Processors

 

This chapter covers

  • Learning some core ideas about stream analytics
  • Querying data streams created by Fluent Bit
  • Exploring the SQL-like syntax provided for streaming analytics.
  • Creating new outputs based on Fluent Bit inputs.

Let’s start by quickly orienting ourselves in the Fluent Bit landscape by seeing how this chapter fits into our technology landscape.

Figure 8.1 Representation of Fluent Bit from a logical architecture perspective, highlighting which aspect of Fluent Bit this chapter will address. Here, we’re particularly interested in the buffer and the stream processing that depends upon the buffer’s capabilities.

Figure 8.1 gives us some immediate insights into what we can do with streaming, with the flow going back to an input. Similarly, we can also output stream processing results to the relevant output plugins.

8.1 Key ideas

8.2 Basic query

8.3 Stream Processing windows

8.3.1 Hopping windows

8.3.2 Tumbling windows

8.3.3 Window durations

8.3.4 Which window to use?

8.4 Selecting multiple attributes and naming

8.5 Streams vs Tags

8.5.1 Applying the STREAM keyword

8.6 Creating Streams

8.7 Chaining and creating new streams

8.7.1 Demonstrating Untagged Stream flow behavior

8.8 Typical use cases for streaming

8.8.1 Forecasting

8.8.2 Intermittent error tolerance

8.8.3 Spurious data values

8.8.4 Detecting absence of events

8.8.5 Stream processing complexity of cross-referencing streams

8.9 Summary