Index lifecycle management

Index lifecycle management (ILM) automates what would otherwise be manual index maintenance. A lifecycle policy declares phases, conditions, and actions; Elasticsearch executes those actions when an index reaches a configured age, size, or document-count threshold. This is especially useful for time-series data such as application or database logs, whose indexes continually grow. section 6.9 section 6.9.2

An index can move through hot, warm, cold, frozen, and delete phases. Hot indexes remain open for searching and indexing. Warm and cold indexes are read-only, allowing older data to be handled differently from actively written data. section 6.10

A policy is applied in two steps:

• Define the lifecycle policy through the _ilm/policy/<policy_name> API, specifying phases, minimum ages, and actions. section 6.9.2
• Associate the policy with an index through the index.lifecycle.name setting, causing the index to transition according to that policy. section 6.9.2

PUT _ilm/policy/hot_delete_policy    #1
{
  "policy": {                        #2
    "phases": {
      "hot": {                       #3
        "min_age": "1d",             #4
        "actions": {                 #5
          "set_priority": {          #6
            "priority": 250
          }
        }
      }, 
      "delete": {                    #7
        "actions": {
          "delete" : { }
        }
      }
    }
  }
}

A minimal policy keeps an index in the hot phase for one day, assigns recovery priority, and then deletes it. With no delete-phase minimum age, deletion begins immediately after hot-phase actions finish. section 6.9.2

PUT hot_delete_policy_index
{
  "settings": {         #1
    "index.lifecycle.name":"hot_delete_policy" # Name of the policy
  }
}

Attach the named policy when creating an index by setting index.lifecycle.name. The policy then controls the index lifecycle rather than requiring manual intervention. section 6.9.2

Rollover creates a new write index when any configured limit is reached. For example, a hot policy can roll over at one day, 10,000 documents, or 10 GB. A rollover alias identifies the current writable index, while an index template can automatically apply the policy and alias settings to matching future indexes. section 6.9.3 section 6.9.3

PUT _ilm/policy/hot_warm_delete_policy
{
  "policy": {
    "phases": {
      "hot": { 
        "min_age": "1d",        #1
        "actions": {
          "rollover": {         #2
            "max_size": "40gb",
            "max_age": "6d"
          },
          "set_priority": {     #3
            "priority": 50
          }
        }
      },
      "warm": { 
        "min_age": "7d",        #4
        "actions": {
          "shrink": {           #5
            "number_of_shards": 1
          }
        }
      },
      "delete": {               #6
        "min_age": "30d",       #7
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

A longer retention policy can roll over a hot index at 40 GB or six days, shrink it to one shard in warm storage after seven days, and permanently delete it after 30 days. Back up data before a delete action because deletion is irreversible. section 6.9.3

Use ILM when growing indexes need predictable retention and resource management. Start with a small policy, verify rollover aliases and template settings, and treat delete actions cautiously because they permanently remove data. section 6.9 section 6.9.3