liveBook · Manning

Random input plugin

As explained in Logs and Telemetry

A random input plugin is a Fluent Bit input component that generates records containing predefined fields with randomly generated values.

The Fluent Bit random input plugin generates records with a predefined JSON structure containing a randomly generated integer, such as {"rand_value"=>9130800433333918836}. It behaves like the dummy input plugin, except the payload is generated rather than supplied. Use it as a synthetic event source when a pipeline needs repeatable input without an external producer. section 8.3 section 10.8

Configure the plugin with a tag and emission interval. A one-second interval tagged random-num creates a steady stream of rand_value records; a five-second service flush allows multiple records to accumulate in the buffer. section 8.3

Select generated values with a stream task

A stream processor can read rand_value from the tagged input with SELECT rand_value FROM TAG:'random-num';. Define the SQL-like expression in a streams file, reference that file through streams_file, then run the random input while directing ordinary output to the null plugin. SELECT results are sent to stdout. section 8.3

Listing 8.2 Config using query stream process: query-streamer.con
[SERVICE]
  flush 5
  log_level info
  streams_file ./query.conf    #1

[INPUT]
  name random              #2
  interval_sec 1
  tag random-num

[OUTPUT]                 #3
  match *
  name null

#[OUTPUT]              #4
#  match *
#  name stdout
#  json_date_format iso8601

This configuration combines a five-second flush, the referenced stream-definition file, a random input every second, the random-num tag, and null output. section 8.3

The generated field can drive several stream-processing checks:

  • Select the source tag alongside rand_value with record_tag(). section 8.3
  • Aggregate generated values with sum(rand_value) and attach processing time with now(). section 8.3
  • Filter and count values with a WHERE clause, count(rand_value), and unix_timestamp(). section 8.3
  • Combine the random input with a filter to create a synthetic source for testing an output plugin. section 10.8

How Logs and Telemetry covers it

How deep it goes
Defines generated records and configuration, then demonstrates tagged SELECT, aggregation, filtering, and time functions.
Examples
Uses random-num records emitted every second, SELECT queries, aggregates, filters, null output, and an output-plugin test source.
What it assumes
Assumes familiarity with Fluent Bit configuration and SQL-like SELECT expressions while showing the required tag and stream-task structure.
What you can do after
Operate/configure it to generate tagged random records for local stream-query and output-plugin tests.
How it connects
Connects random input to dummy input, tags, stream processors, SQL-like expressions, filters, and output plugins.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Logs and Telemetry. The section links point to the parts of the book it draws on.