liveBook · Manning

File output plugin

As explained in Logs and Telemetry

A file output plugin is a Fluent Bit output extension that writes processed events to files.

A file output plugin is a Fluent Bit output extension that writes processed events to files. File outputs provide a broadly applicable destination for local storage, comparison, and operational workflows, while stdout and null outputs help develop or troubleshoot configurations. section 5.1

Configure the destination and representation through the output plugin settings:

  • Set file to choose the filename; otherwise, Fluent Bit uses the tag value. Set path explicitly for production deployments. section 5.5
  • Use plain to write events unmodified, csv with a configurable delimiter, ltsv with label and value delimiters, msgpack, or template formatting. section 5.5
  • Define multiple file outputs when the same matched source must be compared in different formats; give each output a distinct filename. section 5.5
  • Use Fluent Bit as the single filesystem writer to avoid application-level file write collisions, centralize enrichment, and restrict filesystem write permissions. section 5.5

Compare several file representations

An HTTP input tagged http-in can match file outputs writing output1.csv with tab delimiters, output1.ltsv, output1.txt in plain format, output1.msgpack, and a templated text file. A stdout JSON-lines output can provide a concurrent inspection view. section 5.5

Listing 5.5 Different file formats: simple-file.conf
[SERVICE]
  flush 1
  log_level error

[INPUT]
  name http    
  listen 0.0.0.0
  port 9881
  tag http-in

[OUTPUT]        #1
  match *
  name stdout
  json_date_key received-date
  json_date_format iso8601
  format json_lines




[OUTPUT]
  match *
  name file       #2
  file output1.csv          #3
  format csv        #4
  delimiter \t      #5

[OUTPUT]
  match *
  name file
  file output1.ltsv
  format ltsv          #6
  label_delimiter space       #7
  delimiter comma           #8

[OUTPUT]
  match *
  name file
  file output1.txt
  format plain

[OUTPUT]
  match *
  name file
  file output1.msgpack
  format msgpack           #9

[OUTPUT]
  match *
  name file
  format template           #10
  file output1.templated.txt
  template addressing={hello} value={val} --- {time}     #11

Inspect the complete configuration showing HTTP input, wildcard matching, stdout inspection, and five file-format variants. section 5.5

How Logs and Telemetry covers it

How deep it goes
Defines filenames, paths, matching, and five formatting modes through configuration examples without descending into plugin internals.
Examples
Demonstrates HTTP input with wildcard matching and CSV, LTSV, plain, MessagePack, and template file outputs.
What you can do after
Operate/configure Fluent Bit file destinations with explicit paths, filenames, matching rules, delimiters, and output formats.
How it connects
Connects output routing with stdout outputs, null outputs, tags, file formats, data enrichment, and filesystem permissions.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Logs and Telemetry. The section links point to the parts of the book it draws on.