Index Lifecycle Management (ILM) in Elasticsearch is a feature for managing the lifecycle of indices. It helps users optimize storage resource utilization and automatically perform operations such as index creation, optimization, migration, and deletion.
The primary goal of ILM is to automate and optimize the index management process. By defining a set of rules (policies), we can control the entire lifecycle of an index from creation to final deletion. These rules can be triggered based on index age, size, or other conditions.
The ILM workflow is generally divided into four stages:
-
Hot Stage - In this stage, data is frequently written to the index. Typically, indices in the hot stage are stored on high-performance hardware for fast writing and querying.
-
Warm Stage - When an index no longer requires frequent updates but still needs to be queried, it transitions to the warm stage. In this stage, optimizations such as reducing the number of replicas or adjusting the index shard strategy may be performed to reduce storage resource usage.
-
Cold Stage - In the cold stage, the index is no longer frequently queried. The data remains online but can be migrated to lower-cost storage.
-
Delete Stage - Finally, when the data is no longer needed, the index can be automatically deleted to free up resources.
Use Case: In a news website's logging system, the latest click data needs to be frequently accessed and analyzed, so this data is initially placed in the hot stage upon creation. Over time, data from one week ago no longer requires frequent access and is automatically moved to the warm stage, where optimizations such as reducing replicas are performed. After one month, older data is moved to the cold stage, stored on lower-cost, slower-access devices. Finally, when data exceeds a certain time (e.g., three months), it is automatically deleted.
Through ILM, Elasticsearch helps users automatically manage data cost-effectively while maintaining data access performance.