liveBook · Manning

Lifecycle policy

As explained in Elasticsearch in Action, Second Edition

An Elasticsearch lifecycle policy is a set of rules that controls automatic index management, lifecycle phases, transitions, and actions.

A lifecycle policy automates index management as data grows. For time-series logs, it can roll an index over when a new day begins, when it reaches a document count, or when it reaches a size limit instead of relying on scheduled manual work. section 6.9 section 6.0

A policy defines lifecycle phases, the minimum age for a phase, and actions to run when its conditions are met. Attach the policy to an index through the index.lifecycle.name setting so Elasticsearch can move the index through its configured lifecycle. section 6.9.2 section 6.9.3

Configure a lifecycle policy in two core steps:

  • Define the policy with the _ilm/policy/<policy_name> API. A simple policy can keep an index in the hot phase for one day, set its recovery priority, and then delete it. section 6.9.2
  • Associate the policy with an index or an index template using index.lifecycle.name. Template association applies the policy to matching indexes automatically. section 6.9.2 section 6.9.3
  • For rollover, also configure index.lifecycle.rollover_alias and create an initial numbered index whose alias is marked is_write_index: true. section 6.9.3

Roll over MySQL log indexes

A hot-phase policy can roll over mysql-* indexes when any limit is reached: one day of age, 10,000 documents, or 10 GB. Put the policy and rollover alias in a matching index template, then create mysql-index-000001 with mysql-logs-alias as its write alias. When a limit is met, Elasticsearch creates the next numbered index and remaps the alias. section 6.9.3

Expand from hot to warm and delete

A multi-phase policy can roll over hot indexes after 40 GB or six days, lower their recovery priority, shrink them to one shard after seven days in warm, and delete them after 30 days. This makes retention and resource changes explicit lifecycle actions. section 6.9.3

Figure 6.11 Lifecycle of an index

Use the lifecycle diagram to orient phase transitions: indexes begin in hot and may remain there or move to later phases; deletion is the permanent final stage. section 6.9.1

How Elasticsearch in Action, Second Edition covers it

How deep it goes
Walks through policy definition, index and template association, rollover aliases, and a hot-warm-delete configuration.
Examples
Uses daily log indexes, a hot-delete policy, MySQL rollover indexes, and a hot-warm-delete policy.
What it assumes
Assumes familiarity with manually creating and deleting indexes while introducing policy definition and association steps.
What you can do after
Build it from scratch by defining an ILM policy and attaching it to an index or matching index template.
How it connects
Connects index lifecycle management with index templates, rollover aliases, write indexes, shard shrinking, and snapshots.

Also explained in

Compare how 3 books explain lifecycle policy →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Elasticsearch in Action, Second Edition. The section links point to the parts of the book it draws on.