Centralized logging system

As explained in Kafka for Architects

A centralized logging system collects log records from multiple applications or services into a common system for storage, search, and analysis.

Distributed services make local text files insufficient for tracing events and locating failures. Centralized logging provides a unified, real-time view across applications. section 5.1.3

Applications emit standardized log events to Kafka, which processes them before delivery to Elasticsearch or OpenSearch for indexing and analysis. Kibana or OpenSearch Dashboards provides visual access. section 5.1.3

Customer 360 ODS

Multiple services using different languages and frameworks emit consistently formatted events. Kafka routes them to Elasticsearch, where operators search records through its REST API or Kibana. section 5.1.3

The design emphasizes:

• Standardizing event collection across heterogeneous applications. section 5.1.3
• Indexing large log volumes quickly for complex searches and analysis. section 5.1.3

Figure 5.6 Conceptual flow for log data collection: log data is sent from the application to Kafka for processing, then indexed in Elasticsearch, and finally visualized in Kibana.

Follow the conceptual flow from applications through Kafka to Elasticsearch and finally Kibana. section 5.1.3

How Kafka for Architects covers it

How deep it goes

Explains the end-to-end data flow and architectural rationale but stops short of implementation details.

Examples

Uses a Customer 360 ODS with heterogeneous services and an application-to-Kafka-to-Elasticsearch-to-Kibana flow.

What it assumes

Assumes distributed services and teaches why local text files become inadequate as architectures grow more complex.

What you can do after

Operate unified log oversight and investigate events through indexed search and visual interfaces.

How it connects

Connects microservices, Kafka, Elasticsearch, OpenSearch, Kibana, OpenSearch Dashboards, and REST APIs.

liveBookmanning.com

Read Kafka for Architects on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Learn Docker in a Month of Lunches, Second EditionCentralized logging is a logging architecture in which logs from multiple applications or containers are collected and managed in a shared system.
• 

  liveBookmanning.com
  ASP.NET Core in Action, 3eCentralized logging is a logging arrangement in which application instances send their logs to a shared location separate from the applications.

Compare how 3 books explain centralized logging system →

Related concepts

• fluentduses · Fluentd is an open-source data collector that gathers, processes, transforms, and routes logs and other machine-generated event data through pluggable integrations.
• monitoringaddresses · Monitoring is the ongoing observation and measurement of a system’s behavior, state, or operational signals to assess its condition and detect changes or problems.
• log aggregationuses · Log aggregation is the collection and consolidation of log records from multiple sources into a common destination for storage, search, or analysis.
• security information and event managementimplemented by · Security information and event management is a class of security systems that centrally collects, correlates, and analyzes security events and log data to support monitoring and response.

This explainer was generated by Manning's knowledge graph from the text of Kafka for Architects. The section links point to the parts of the book it draws on.