liveBook · Manning

Pipeline aggregation

As explained in Elasticsearch in Action, Second Edition

A pipeline aggregation computes results from the output of other aggregations.

Pipeline aggregations answer questions that require a second calculation over aggregation results. For example, daily cappuccino sales can become a running total, rather than remaining separate daily values. section 13.5

Unlike metric and bucket aggregations, which calculate from documents or group them into buckets, a pipeline aggregation consumes the output of another aggregation. section 13.0 section 13.6

Choose the pipeline type according to where its input aggregation sits:

  • Parent pipelines work on a child aggregation and add resulting buckets or aggregations to the existing buckets. section 13.5.1
  • Sibling pipelines calculate a new aggregation at the same level as the aggregations supplying their input. section 13.5.1
  • Use buckets_path to identify the metric or bucket aggregation whose output feeds the pipeline. section 13.5.3

Running cappuccino sales total

After indexing coffee_sales data containing dates and product sales, place a cumulative_sum pipeline after the cappuccino_sales aggregation. The pipeline uses buckets_path to refer to cappuccino_sales, retaining the accumulated total for each successive day. Other useful choices include derivative for a histogram metric, moving_function for a sliding window, and max_bucket for the highest-valued bucket. section 13.5.2 section 13.5.3 section 13.5.4

Listing 13.21 Indexing data using the _bulk API
PUT coffee_sales/_bulk
{"index":{"_id":"1"}}
{"date":"2022-09-01","sales":{"cappuccino":23,"latte":12,"americano":9,
 "tea":7},"price":{"cappuccino":2.50,"latte":2.40,"americano":2.10,
 "tea":1.50}}
{"index":{"_id":"2"}}
{ "date":"2022-09-02","sales":{"cappuccino":40,"latte":16,"americano":19,
 "tea":15},"price":{"cappuccino":2.50,"latte":2.40,"americano":2.10,
 "tea":1.50}}

Inspect the coffee_sales documents used to demonstrate pipeline aggregation inputs. section 13.5.2

Table 13.1 Parent pipeline aggregations (view table figure)

Name

Description

Buckets script (buckets_script)

Executes a script on multi-bucket aggregations

Bucket selector (bucket_selector)

Executes a script to select the current bucket for its place in the multi-bucket aggregation

Bucket sort (bucket_sort)

Sorts the buckets

Cumulative cardinality (cumulative_cardinality)

Checks recently added unique (cumulative cardinality) values

Cumulative sum (cumulative_sum)

Finds the cumulative sum of a metric

Derivative (derivative)

Finds the derivative of a metric in a histogram or date histogram

Inference (inference)

Finds the inference on a pretrained model

Moving function (moving_function)

Executes a custom script on a sliding window

Moving percentiles (moving_percentiles)

Similar to moving_function, except it calculates in percentiles

Normalize (normalize)

Calculates the normalized value of a given bucket

Serial difference (serial_diff)

Calculates the serial difference on a metric

Browse the available parent pipeline aggregations and their purposes. section 13.5.4

Table 13.2 Sibling pipeline aggregations (view table figure)

Name

Description

Average (avg_bucket)

Calculates the average value of the sibling metric

Bucket count (bucket_count_ks_test)

Calculates the Kolmogorov–Smirnov statistic over a distribution

Bucket correlation (bucket_correlation)

Executes a correlation function

Change point (change_point)

Detects the dips and changes in a metric

Extended stats (extended_stats)

Calculates multiple statistical functions

Max bucket (max_bucket)

Finds the maximum-valued bucket

Min bucket (min_bucket)

Finds the minimum-valued bucket

Percentiles bucket (percentiles_bucket)

Calculates the percentiles of a metric

Stats bucket (stats_bucket)

Calculates common stats for a metric

Sum bucket (sum_bucket)

Calculates the sum of a metric

Browse the available sibling pipeline aggregations and their purposes. section 13.5.4

How Elasticsearch in Action, Second Edition covers it

How deep it goes
Explains parent and sibling types plus buckets_path, then demonstrates common pipelines without documenting every available aggregation.
Examples
Demonstrates coffee sales, daily cappuccino totals, cumulative_sum, buckets_path, coffee_sales, and parent-versus-sibling aggregation layouts.
What it assumes
Assumes familiarity with metric and bucket aggregations while establishing pipeline aggregation behavior through examples.
What you can do after
Build it from scratch by chaining metric or bucket aggregations and directing their output with buckets_path.
How it connects
Connects metric aggregations, bucket aggregations, histograms, date histograms, cumulative sums, derivatives, moving averages, and bucket statistics.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Elasticsearch in Action, Second Edition. The section links point to the parts of the book it draws on.