Logging pipeline

As explained in Cloud Observability in Action

A logging pipeline is a sequence of components that collects, routes, processes, stores, and presents log records.

A logging pipeline is a sequence of components that collects, routes, processes, stores, and presents log records. A containerized example sends ho11y logs through Fluentd and Fluent Bit into OpenSearch, where OpenSearch Dashboards supports querying and visualization. section 4.1.1

The example assigns distinct operational roles:

• ho11y generates log signals and exposes an HTTP API. section 4.1.1
• Fluentd provides the Docker logging driver and forwards records to Fluent Bit. section 4.1.1
• Fluent Bit receives records through its Forward input and routes matching records to OpenSearch. section 4.1.1
• OpenSearch stores logs, while OpenSearch Dashboards provides search, discovery, and visualization. section 4.1.1

Run and inspect the pipeline

Docker Compose starts the components with `docker-compose -f docker-compose.yaml up`. A loop repeatedly calls ho11y’s HTTP endpoint, generating records that can be selected through an OpenSearch index pattern and inspected in the Discover view. section 4.1.1

Figure 4.1 Example logging pipeline using Fluentd and Fluent Bit

Trace the end-to-end path from the signal source through Fluentd and Fluent Bit to OpenSearch and its dashboard. section 4.1.1

How Cloud Observability in Action covers it

How deep it goes

Explains the pipeline component by component and shows Docker Compose and Fluent Bit configuration, stopping short of backend detail.

Examples

Uses ho11y containers, Docker Compose files, Fluent Bit configuration, generated HTTP traffic, and OpenSearch index patterns.

What you can do after

Operate and configure a Docker Compose pipeline that routes container logs into OpenSearch for dashboard-based inspection.

How it connects

Connects log routing with Fluentd, Fluent Bit, Docker Compose, OpenSearch, and OpenSearch Dashboards.

liveBookmanning.com

Read Cloud Observability in Action on liveBook Get the book at manning.com

Related concepts

• pipelinekind of · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.

This explainer was generated by Manning's knowledge graph from the text of Cloud Observability in Action. The section links point to the parts of the book it draws on.