liveBook · Manning

Output plugin

As explained in Logs and Telemetry

An output plugin is a Fluent Bit extension that retrieves processed events and stores or forwards them to a destination.

An output plugin takes processed events from the Fluent Bit buffer and delivers them to a destination. It can store data locally, forward it to another Fluent Bit or Fluentd instance, or send it to a third-party system for storage, analysis, visualization, or onward processing. Outputs complete the pipeline after inputs collect events and filters or stream processing transform them. section 1.2.2 section 5.0

Destinations range from local console, null, and file outputs to network protocols, self-managed data stores, and cloud services. Most outputs have network behavior; configuration may therefore include authentication and mappings that label events or identify the service being sent. OpenTelemetry, HTTP, WebSockets, Prometheus formats, and Fluent Bit forward are examples of protocol or format-oriented integration paths. section 5.1

Use output choices to match an operational need:

  • Use stdout or null while developing or investigating pipeline configuration; null still exposes throughput through plugin metrics. section 5.1 section 5.3.2
  • Use file output for a broadly applicable local destination. section 5.1
  • Use protocol, datastore, or cloud outputs to forward logs, traces, and metrics to systems that query, analyze, or visualize them. section 5.0 section 5.1

Output plugins report OK, error, or retry results. Fluent Bit can pause and retry before giving up, using service-wide scheduler.base and scheduler.cap values to bound retry delays; an output retry_limit controls retry attempts. This matters even for apparently local destinations, because enterprise and Kubernetes storage can depend on networks. section 5.2.1

Verify a pipeline with null output

Configure CPU, network-interface, and memory inputs with a null output matching all tags. Run Fluent Bit, then query its HTTP metrics endpoint. Input record and byte totals should correspond closely to the null output totals, subject to records being processed while metrics are retrieved. section 5.3.2

A custom output plugin consumes buffered data during its flush callback and sends it to its target. Context-aware flush and exit methods support separate configurations and independent cleanup for multiple instances of one plugin type. In the Go database example, the callback receives MessagePack through C pointers, decodes records, converts values to suitable Go types, and inserts them using configuration-derived parameters. section 10.5.2 section 10.6.4

Figure 10.4 The life cycle of a Fluent Bit output plugin. Each rectangle block represents an implementable method. Here, we have output flush with or without a context object and a corresponding exit with or without context. Each method we implement is triggered by the Fluent Bit core, as shown with the full-width shapes. We can see the logical life cycle with clarity on the invocation source.
figure

The lifecycle diagram distinguishes flush and exit callbacks with and without context, and shows that Fluent Bit core triggers each implemented method. section 10.5.2

How Logs and Telemetry covers it

How deep it goes
Explains destination categories, retry controls, metrics validation, and custom flush lifecycle down to MessagePack decoding through C pointers.
Examples
Uses null output with CPU, network, and memory inputs, HTTP metrics, retrying forward output, and a Go database plugin.
What it assumes
Assumes familiarity with Fluent Bit inputs and basic filters before progressing from I/O outputs to more advanced processing.
What you can do after
Operate/configure outputs for local validation, external delivery, and retry behavior; evaluate plugin throughput through HTTP metrics.
How it connects
Connects buffers, inputs, filters, stream processing, OpenTelemetry, Prometheus, Fluentd, retries, and MessagePack.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Logs and Telemetry. The section links point to the parts of the book it draws on.