liveBook · Manning

Fluentd

As explained in Logs and Telemetry

Fluentd is an open-source data collector that unifies the collection and routing of logs and other machine data through pluggable integrations.

Fluentd is a dominant observability-data collector in the Fluent ecosystem. Its pluggable integrations make it useful for connecting varied sources and targets, gathering logs and metrics, and preparing data for tools that render or analyze it. section 1.3 section 1.1.2

Fluentd and Fluent Bit are complementary rather than mutually exclusive. They communicate seamlessly, including through the forward plugin, so deployments can blend both tools or migrate gradually. Fluentd remains particularly suited to non-cloud-native environments with broad adapter needs and accessible custom-plugin skills. section 1.3 section 1.4 section 3.9

A Fluentd pipeline can be understood through its configuration directives:

  • `<system>` sets node-wide behavior, such as the log level. section 2.2.1
  • `<source>` defines an input, such as a Dummy source that emits `{"hello": "my world"}`. section 2.2.1
  • `<match *>` routes matching events to an output; the example sends every event to stdout. section 2.2.1

Minimal stdout pipeline

A small Fluentd configuration can set `log_level info`, create a Dummy input, and match all tags to the stdout output. It demonstrates the basic flow: generate an event, match it, and write it to the console. section 2.2.1

Table 2.4 Classic Fluent Bit vs. Fluentd configuration comparison
Classic Fluent Bit
Fluentd
[SERVICE]
    flush      1
    daemon     Off
    log_level  info

# define the Dummy source 
[INPUT]
    name  Dummy
    dummy {"hello": "my world"}


# Accept all log events regardless of tag and write 
# them to the console
[OUTPUT]
    name  stdout
    match *
<system>
    log_level info
</system>

# define the Dummy source
<source>
    @type Dummy
    dummy {"hello": "my world"}
    
</source> # after a directive
# Accept all log events regardless of tag and write them to the console
<match *>
   @type stdout
</match>

Compare Fluentd’s XML-style directives with Fluent Bit’s bracketed configuration syntax when planning coexistence or migration. section 2.2.1

For Kubernetes-centered and containerized deployments, Fluent Bit is described as the lighter choice and has OpenTelemetry support. Fluentd is not therefore automatically legacy: selecting between them depends on architecture, existing adapters, and operational constraints. section 1.4 section 1.3.5

How Logs and Telemetry covers it

How deep it goes
Defines Fluentd’s ecosystem role, deployment fit, interoperability, and basic configuration directives without detailing plugin implementation.
Examples
Demonstrates a Dummy source routed through `<match *>` to stdout and compares its configuration with Fluent Bit.
What it assumes
Assumes no Fluentd knowledge when learning Fluent Bit, while noting that high-level Fluentd knowledge eases comparison.
What you can do after
Apply it via Fluentd directives to define an input and route all matching events to stdout.
How it connects
Relates Fluentd to Fluent Bit, OpenTelemetry, Kubernetes, Istio, CNCF, and the forward plugin.

Also explained in

Compare how 4 books explain fluentd →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Logs and Telemetry. The section links point to the parts of the book it draws on.