liveBook · Manning

Stream task

As explained in Logs and Telemetry

A stream task is a configured Fluent Bit operation that applies a query or processing definition to an event stream or tag.

A stream task is a named Fluent Bit stream-processor definition. It uses a SQL-like expression to read event attributes from a tagged input or a named stream and emit the query result. Define it in a separate streams configuration file, then load that file through the main configuration. section 8.3 section 8.6

A task definition has three essential parts:

  • `[STREAM_TASK]` starts a referenceable task definition. section 8.3
  • `name` assigns the identifier used for the expression. section 8.3
  • `exec` supplies a semicolon-terminated SQL-like query, such as `SELECT rand_value FROM TAG:'random-num';`. section 8.3
Listing 8.1 Query stream processor definition: query.conf
[STREAM_TASK]      #1
  name    selectTask           #2
  exec     SELECT rand_value FROM TAG:'random-num';      #3

This minimal task selects the `rand_value` attribute from events tagged `random-num`. A main configuration can load it with `streams_file`; with ordinary output sent to `null`, the SELECT result is the visible processor output. section 8.3

Use `TAG:'tag-name'` when selecting by tag; quotes accommodate unusual tag characters and tags can use wildcards. Use `STREAM:input.instance` when selecting a specific input stream; stream names do not use quotes and do not support wildcards. Multiple tasks can independently select a record tag and value, calculate `sum(rand_value)`, or filter with `WHERE rand_value > 0`. section 8.3 section 8.6

Switching a task from a tag to a stream

To bind a query to the first random input instance rather than every matching tag, replace `FROM TAG:'random-num'` with `FROM STREAM:random.0`. The resulting task can select values, calculate sums, or count values with a `WHERE` condition. Stream-based tasks can also provide streams to which downstream processors attach. section 8.6

How Logs and Telemetry covers it

How deep it goes
Explains task declarations, SELECT expressions, functions, WHERE filtering, and TAG-versus-STREAM source selection without detailing query internals.
Examples
Demonstrates random-input values queried by tag, then repeats the tasks against `STREAM:random.0`.
What it assumes
Teaches basic task configuration from a random-input example and states no explicit prior requirements.
What you can do after
Operate/configure it by defining named queries in a streams file and selecting an event source by tag or stream.
How it connects
Connects stream processors to Fluent Bit inputs, tags, streams, aliases, SQL-like SELECT queries, and downstream processors.
Cover of Logs and Telemetry
liveBookmanning.com
Read Logs and Telemetry on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Logs and Telemetry. The section links point to the parts of the book it draws on.