Fluentd

As explained in Kafka for Architects

Fluentd is an open-source data collector that gathers, transforms, and routes event and log data.

Fluentd is an open-source data collector used to move application log events into centralized processing and storage architectures. section 5.1.3

A Kafka-based logging flow separates application code from transport:

• Applications emit events through a generic logging framework rather than depending directly on Kafka. section 5.1.3
• Fluentd agents collect log data from files and forward it to Kafka as log events. section 5.1.3
• Kafka Connect moves the data from Kafka to Elasticsearch for centralized indexing. section 5.1.3

Figure 5.7 Sending log data via Kafka to Elasticsearch

Use the architecture figure to inspect the Fluentd-to-Kafka-to-Elasticsearch log pipeline. section 5.1.3

How Kafka for Architects covers it

How deep it goes

Traces the collection path through Fluentd, Kafka, Kafka Connect, and Elasticsearch without detailing collector internals.

Examples

Uses file-based application logs, Fluentd agents, Kafka topics, Kafka Connect, and Elasticsearch as the concrete pipeline.

What it assumes

Assumes application logging and Kafka concepts while separating logging APIs from transport configuration.

What you can do after

Operate/configure a decoupled log pipeline that forwards application events through Kafka into Elasticsearch.

How it connects

Connects generic logging frameworks with Kafka, Kafka Connect, Elasticsearch, Logstash, Fluent Bit, and syslog.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Learn Docker in a Month of Lunches, Second EditionFluentd is an open-source unified logging layer that collects, processes, enriches, filters, and forwards log data.
• 

  liveBookmanning.com
  Logs and TelemetryFluentd is an open-source data collector that unifies the collection and routing of logs and other machine data through pluggable integrations.
• 

  liveBookmanning.com
  Cloud Observability in ActionFluentd is an open-source data collector that unifies the collection and forwarding of logs and other machine data.

Compare how 4 books explain fluentd →

Related concepts

• Elasticsearchuses · Elasticsearch is a distributed search and analytics engine that indexes data for searching, retrieval, and analysis.
• EFK stackused by · The EFK stack is a logging architecture composed of Elasticsearch for storage and search, Fluentd for collection and processing, and Kibana for exploration and visualization.
• FluentBitcontrasts with · Fluent Bit is a lightweight, open-source telemetry agent for collecting, processing, enriching, and forwarding logs and other event data.
• Apache Kafkauses · Apache Kafka is a distributed event-streaming platform for publishing, storing, processing, and consuming streams of records.
• forward input pluginused by · A forward input plugin is a Fluent Bit input component that receives events or log records encoded with the Fluentd Forward protocol.
• Fluentd logs driverused by · The Fluentd logging driver is a Docker logging integration that forwards container log events to a Fluentd service using the Fluent Forward protocol.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.