ElasticSearch备考 -- Manage the index lifecycle (ILM)

一、题目

在集群中，数据首先分布在data_hot节点，rollover 设置max_age:3d, max_docs:5,max_size:50gb, 优先级为100。

max_age:15s, forcemarge 段合并，数据迁移到data_warm节点，副本数为0，优先级为50

max_age:30s, 数据从data_warm迁移到data_cold节点

max_age:60s, 数据删除

运用ilm，创建index template名字为task_ilm_template,匹配模式task_ilm-*，并指定ilm 策略，创建索引task_ilm_00001注意分布情况

二、思考

索引生命周期管理ILM是对索引数据管理的一种自动化方式，主要是通过policy+索引模版结合使用。

通过创建ILM，在policy中可以指定多个阶段，生命周期阶段包括：

Hot: The index is actively being updated and queried.
Warm: The index is no longer being updated but is still being queried.
Cold: The index is no longer being updated and is queried infrequently. The information still needs to be searchable, but it’s okay if those queries are slower.
Frozen: The index is no longer being updated and is queried rarely. The information still needs to be searchable, but it’s okay if those queries are extremely slow.
Delete: The index is no longer needed and can safely be removed.

每个阶段可以制定不同的动作action，action主要有以下：

Rollover: Creates a new write index when the current one reaches a certain size, number of docs, or age.
Shrink: Reduces the number of primary shards in an index.
Force merge: Triggers a force merge to reduce the number of segments in an index’s shards.
Delete: Permanently remove an index, including all of its data and metadata.

通过创建索引模版并配置ILM策略，可以在使用模版创建索引后，对数据起到一个自动化生命周期管理的目的。

三、解题

Step 1、前置工作修改集群

1）集群节点角色配置

node-1：data_hot, data_content
node-2：data_warm, data_content
node-3：data_cold, data_content

2）索引生命周期刷新间隔

对于生产环境非必要配置，主要是为了索引生命周期刷新轮询间隔为1s，在验证时候能快速看到效果。

PUT _cluster/settings
{"persistent": {"indices.lifecycle.poll_interval": "1s"}
}

Step 2、创建ILM策略

可以通过命令行和图形化配置两种方式

【命令行方式】

PUT _ilm/policy/task_test_policy
{"policy": {"phases": {"hot": {"min_age": "0ms","actions": {"set_priority": {"priority": 100},"rollover": {"max_size": "50gb","max_primary_shard_size": "50gb","max_age": "3d","max_docs": 5}}},"warm": {"min_age": "15s","actions": {"forcemerge": {"max_num_segments": 1,"index_codec": "best_compression"},"set_priority": {"priority": 50},"allocate": {"number_of_replicas": 0}}},"cold": {"min_age": "30s","actions": {"readonly": {}}},"delete": {"min_age": "60s","actions": {"delete": {"delete_searchable_snapshot": true}}}}}
}

【图形化方式】

打开kibana在左侧菜单中找到management展开点击stack management

在stack management中找到index lifecycle polices菜单，点击右侧的create ploicy的按钮

配置hot phase

配置 warm phase

配置 cold phase

配置 delete phase，需要在 cold phase中点击删除按钮，会增加一个delete phase

Step 3、创建索引模版配置ILM策略

注意：在创建索引模版的时候要指定lifecycle，也就是刚刚创建的ILM策略。此外，还要指定rollover滚动操作别名，缺少别名创建的index也不能正常执行ILM策略。

【命令行方式】

PUT _index_template/task_ilm_template
{"settings": {"index": {"lifecycle": {"name": "task_test_policy","rollover_alias": "task_ilm"},"routing": {"allocation": {"include": {"_tier_preference": null}}},"number_of_shards": "5","number_of_replicas": "1"}},"mappings": {"properties": {"age": {"type": "integer"},"name": {"type": "text"}}},"aliases": {}
}

【图像界面方式】

在左侧菜单中点击data，在右侧主页面中选择index template 标签页，并点击create template按钮

创建索引模版操作如下

此处需要注意：

1）匹配模式最后通配符要使用“-”中杠

2）需要指定配置，重点需要配置数据节点

创建模版中所需的字段及类型

最后点击Create template按钮

Step 4、创建索引验证

注意这里有两个点

1）需要指定索引的别名，并且可以基于别名进行写入操作

2）数据的批量写入操作是基于别名的

# DELETE task_ilm-000001  
PUT task_ilm-000001 
{"aliases": {"task_ilm": {"is_write_index":true}}
}POST task_ilm/_bulk
{"create":{"_id":1}}
{"name":"key 1","age":1}
{"create":{"_id":2}}
{"name":"key 2","age":2}
{"create":{"_id":3}}
{"name":"key 3","age":3}
{"create":{"_id":4}}
{"name":"key 4","age":4}
{"create":{"_id":5}}
{"name":"key 5","age":5}POST task_ilm/_bulk
{"create":{"_id":6}}
{"name":"key 6","age":6}

创建后索引获取索引配置