Fluentd

As explained in Cloud Observability in Action

Fluentd is an open-source data collector that unifies the collection and forwarding of logs and other machine data.

Fluentd is an open-source data collector that unifies log and machine-data collection with forwarding. Its rich plug-in ecosystem supports flexible routing, but its Ruby-and-C implementation has a memory footprint of dozens of megabytes. section 4.1.1

Use Fluentd when:

• A containerized service needs to send tagged logs through a logging driver to a centralized backend. section 4.1.1
• A large plug-in ecosystem matters more than the smallest possible agent footprint. section 4.1.1

Containerized log pipeline

A ho11y container tags its output as ho11y and uses the Fluentd logging driver. Fluent Bit receives and routes those records to OpenSearch, where OpenSearch Dashboards supports querying and visualization. section 4.1.1

Figure 4.1 Example logging pipeline using Fluentd and Fluent Bit

Trace the signal path from the container and Fluentd logging driver through Fluent Bit to OpenSearch and its dashboard. section 4.1.1

How Cloud Observability in Action covers it

How deep it goes

Defines Fluentd, contrasts its footprint and plug-ins with Fluent Bit, and demonstrates a pipeline without detailing Fluentd configuration internals.

Examples

Uses a ho11y container, tagged records, Fluent Bit routing, and OpenSearch querying in a Docker Compose pipeline.

What it assumes

Assumes familiarity with containers and Docker Compose while deferring detailed OpenSearch treatment.

What you can do after

Operate/configure a containerized log pipeline that forwards tagged application records to a searchable backend.

How it connects

Connects Fluentd with Fluent Bit, Docker Compose, logging drivers, OpenSearch, and OpenSearch Dashboards.

liveBookmanning.com

Read Cloud Observability in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Learn Docker in a Month of Lunches, Second EditionFluentd is an open-source unified logging layer that collects, processes, enriches, filters, and forwards log data.
• 

  liveBookmanning.com
  Logs and TelemetryFluentd is an open-source data collector that unifies the collection and routing of logs and other machine data through pluggable integrations.
• 

  liveBookmanning.com
  Kafka for ArchitectsFluentd is an open-source data collector that gathers, transforms, and routes event and log data.

Compare how 4 books explain fluentd →

Related concepts

• Elasticsearchuses · Elasticsearch is a distributed search and analytics engine that indexes data for searching, retrieval, and analysis.
• EFK stackused by · The EFK stack is a logging architecture composed of Elasticsearch for storage and search, Fluentd for collection and processing, and Kibana for exploration and visualization.
• FluentBitcontrasts with · Fluent Bit is a lightweight, open-source telemetry agent for collecting, processing, enriching, and forwarding logs and other event data.
• Apache Kafkauses · Apache Kafka is a distributed event-streaming platform for publishing, storing, processing, and consuming streams of records.
• forward input pluginused by · A forward input plugin is a Fluent Bit input component that receives events or log records encoded with the Fluentd Forward protocol.
• Fluentd logs driverused by · The Fluentd logging driver is a Docker logging integration that forwards container log events to a Fluentd service using the Fluent Forward protocol.

This explainer was generated by Manning's knowledge graph from the text of Cloud Observability in Action. The section links point to the parts of the book it draws on.