liveBook · Manning

Tail input plugin

As explained in Logs and Telemetry

A tail input plugin is a Fluent Bit input component that reads appended records from files, typically following appended content.

The Fluent Bit tail input plugin reads records from files. Like Linux `tail -f`, it can begin at a file’s head and follow later appends. Configure file locations and, when restart continuity matters, DB-prefixed settings that record reading progress. section 3.5

A basic file-ingestion setup chooses how the input is located and consumed:

  • Set `path` to select the file; use `exclude_path` to avoid unwanted neighbors such as compressed archives. section 3.5.1
  • Set a `tag` explicitly when each line is handled as an unstructured string. section 3.5.1
  • Use `read_from_head true` to ingest existing contents rather than only new appends, and `exit_on_eof true` for a bounded file-read run. section 3.5.1
Listing 3.4 Illustrating a file tail: basic-file-read.conf
[SERVICE]
  flush 1

[INPUT]
  name tail     #1

  path ./basic-file.txt       #2

  read_from_head true      #3

  exit_on_eof true        #4

  tag basic-file      #5

[OUTPUT]
  name stdout
  match *

This minimal configuration reads `./basic-file.txt` from the start, labels records `basic-file`, writes them to stdout, and exits at end of file. section 3.5.1

Parsing can happen directly at the tail input. Supplying a parser avoids serializing events to MessagePack and deserializing them again for a separate parse step, reducing processing and buffer overhead—especially when buffers use file storage. section 6.11

Parse while reading

A tail input can read `./codedJSON.log`, tag records `codedFeed`, and set `parser myJSONDecodedUTF`. The parsed events then go directly to stdout without a separate parsing filter. section 6.11

For application stdout that would otherwise be piped directly into Fluent Bit, writing to a log file and tailing it can be an alternative. Keep parsing simple, use buffering where needed, and use log rotation when files grow unwieldy. section 3.3.2

How Logs and Telemetry covers it

How deep it goes
Explains file selection, tagging, head reads, EOF behavior, persisted progress, and input-level parser optimization.
Examples
Uses local text and coded JSON files, stdout output, a LogSimulator, and a minimal Fluent Bit configuration.
What it assumes
Assumes a runnable Fluent Bit setup and accessible log files; begins with unstructured line ingestion.
What you can do after
Apply it via Fluent Bit configuration to read files, control starting position, label records, and parse at input.
How it connects
Connects file ingestion with tags, parsers, MessagePack, buffer storage, Kubernetes logs, and logrotate.
Cover of Logs and Telemetry
liveBookmanning.com
Read Logs and Telemetry on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Logs and Telemetry. The section links point to the parts of the book it draws on.