Logstash

As explained in Elasticsearch in Action, Second Edition

Logstash is an open-source data-processing engine that extracts, transforms, enriches, and routes event data between sources and destinations.

Logstash is an open-source data-processing engine for moving event data from multiple sources to different destinations. It extracts, transforms, enriches, and routes events through real-time pipelines governed by preconfigured parsing rules. section 1.4.2

A typical Elasticsearch ingestion setup uses Logstash when source data needs reshaping before indexing. Database records, files, logs, metrics, and real-time application events can enter the pipeline and emerge as searchable Elasticsearch data. section 3.1.1

Logstash fits into ingestion workflows by:

• Reading from sources such as files, HTTP, JMS, Kafka, Amazon S3, Twitter, databases, and application event streams. section 1.4.2 section 3.1.1
• Transforming and enriching records whose shape does not yet match Elasticsearch’s indexing needs. section 3.1.1
• Sending processed events to destinations such as Elasticsearch and supporting search, debugging, analysis, and storage. section 1.4.2 section 3.1.1

Database-to-search pipeline

An organization can fetch records from its authoritative database in batches or near real time, use Logstash to transform and enrich them, and index the resulting data into Elasticsearch for efficient search and analytics. section 3.1.1

Figure 1.3 The Elastic Stack: Beats, Logstash, Elasticsearch, and Kibana

Use the Elastic Stack diagram to trace Logstash’s position between incoming data sources and Elasticsearch, alongside Beats and Kibana. section 1.4.2

How Elasticsearch in Action, Second Edition covers it

How deep it goes

Defines sources, transformations, destinations, and real-time pipeline behavior without detailing configuration syntax or internals.

Examples

Uses database records, files, logs, metrics, application events, Kafka, HTTP, JMS, Amazon S3, and Twitter as pipeline inputs.

What it assumes

Assumes familiarity with data sources and Elasticsearch indexing while framing Logstash’s role at a high level.

What you can do after

Operate Logstash as an ingestion layer that transforms and enriches source data before Elasticsearch indexing.

How it connects

Connects Logstash with Elasticsearch, Beats, Kibana, ETL, data ingestion, search, analytics, and real-time streaming.

liveBookmanning.com

Read Elasticsearch in Action, Second Edition on liveBook Get the book at manning.com

Related concepts

• ELK stackpart of · The ELK stack is a collection of Elasticsearch, Logstash, and Kibana components for ingesting, processing, searching, analyzing, and visualizing data.
• Elasticsearchuses · Elasticsearch is a distributed search and analytics engine that indexes data for searching, retrieval, and analysis.
• ETLimplements · ETL is a data-integration process that extracts data from source systems, transforms it, and loads the result into a target system.
• FluentBitused by · Fluent Bit is a lightweight, open-source telemetry agent for collecting, processing, enriching, and forwarding logs and other event data.

This explainer was generated by Manning's knowledge graph from the text of Elasticsearch in Action, Second Edition. The section links point to the parts of the book it draws on.