场景:kafka配置文件log.dirs增加了几个目录,但是新目录没有分区数据写入,所以打算进行重分区一下。
1.生成迁移计划
进入kafka/bin目录
新建 topic-reassign.json,把要重分区的topic按下面格式写。
{
"topics": [{
"topic": "VOLTE_GM"
},{
"topic": "VOLTE_GM_MEDIA"
},{
"topic": "VOLTE_MW"
},{
"topic": "VOLTE_RX"
},{
"topic": "VOLTE_SH"
},{
"topic": "VOLTE_SV"
}],
"version": 1
}
执行命令
./kafka-reassign-partitions.sh --zookeeper kafka001:2181/kafka --generate --topics-to-move-json-file topic-reassign.json --broker-list 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39
broker-list 是kafka配置文件中的 broker.id
输出:
Current partition replica assignment:当前分区策略,建议保存,回退的时候会用到。
Proposed partition reassignment configuration:建议分区策略
Current partition replica assignment
{"version":1,"partitions":[{"topic":"VOLTE_GM","partition":1,"replicas":[35,18],"log_dirs":["any","any"]},{"topic":"VOLTE_GM_MEDIA","partition":2,"replicas":[0,19],"log_dirs":["any","any"]},{"topic":"VOLTE_RX","partition":0,"replicas":[33,2],"log_dirs":["any","any"]},{"topic":"VOLTE_SV","partition":1,"replicas":[10,2],"log_dirs":["any","any"]},{"topic":"VOLTE_SH","partition":0,"replicas":[14,13],"log_dirs":["any","any"]},{"topic":"VOLTE_GM","partition":2,"replicas":[36,19],"log_dirs":["any","any"]},{"topic":"VOLTE_SV","partition":2,"replicas":[11,3],"log_dirs":["any","any"]},{"topic":"VOLTE_GM_MEDIA","partition":1,"replicas":[39,18],"log_dirs":["any","any"]},{"topic":"VOLTE_SH","partition":2,"replicas":[16,15],"log_dirs":["any","any"]},{"topic":"VOLTE_SH","partition":1,"replicas":[15,14],"log_dirs":["any","any"]},{"topic":"VOLTE_MW","partition":1,"replicas":[38,23],"log_dirs":["any","any"]},{"topic":"VOLTE_MW","partition":2,"replicas":[39,24],"log_dirs":["any","any"]},{"topic":"VOLTE_GM_MEDIA","partition":0,"replicas":[38,17],"log_dirs":["any","any"]},{"topic":"VOLTE_RX","partition":2,"replicas":[35,4],"log_dirs":["any","any"]},{"topic":"VOLTE_MW","partition":0,"replicas":[37,22],"log_dirs":["any","any"]},{"topic":"VOLTE_GM","partition":0,"replicas":[34,17],"log_dirs":["any","any"]},{"topic":"VOLTE_RX","partition":1,"replicas":[34,3],"log_dirs":["any","any"]},{"topic":"VOLTE_SV","partition":0,"replicas":[9,1],"log_dirs":["any","any"]}]}Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"VOLTE_RX","partition":0,"replicas":[28,21],"log_dirs":["any","any"]},{"topic":"VOLTE_SH","partition":0,"replicas":[1,25],"log_dirs":["any","any"]},{"topic":"VOLTE_GM_MEDIA","partition":0,"replicas":[36,19],"log_dirs":["any","any"]},{"topic":"VOLTE_SH","partition":2,"replicas":[3,27],"log_dirs":["any","any"]},{"topic":"VOLTE_MW","partition":0,"replicas":[15,13],"log_dirs":["any","any"]},{"topic":"VOLTE_RX","partition":2,"replicas":[30,23],"log_dirs":["any","any"]},{"topic":"VOLTE_GM","partition":1,"replicas":[12,32],"log_dirs":["any","any"]},{"topic":"VOLTE_SV","partition":0,"replicas":[35,25],"log_dirs":["any","any"]},{"topic":"VOLTE_MW","partition":2,"replicas":[17,15],"log_dirs":["any","any"]},{"topic":"VOLTE_GM_MEDIA","partition":2,"replicas":[38,21],"log_dirs":["any","any"]},{"topic":"VOLTE_SV","partition":2,"replicas":[37,27],"log_dirs":["any","any"]},{"topic":"VOLTE_SH","partition":1,"replicas":[2,26],"log_dirs":["any","any"]},{"topic":"VOLTE_RX","partition":1,"replicas":[29,22],"log_dirs":["any","any"]},{"topic":"VOLTE_GM","partition":0,"replicas":[11,31],"log_dirs":["any","any"]},{"topic":"VOLTE_MW","partition":1,"replicas":[16,14],"log_dirs":["any","any"]},{"topic":"VOLTE_GM_MEDIA","partition":1,"replicas":[37,20],"log_dirs":["any","any"]},{"topic":"VOLTE_SV","partition":1,"replicas":[36,26],"log_dirs":["any","any"]},{"topic":"VOLTE_GM","partition":2,"replicas":[13,33],"log_dirs":["any","any"]}]}
将标红的json保存到topic-todo.json
2.执行迁移命令
./kafka-reassign-partitions.sh --zookeeper kafka001:2181/kafka --reassignment-json-file topic-todo.json --execute --throttle 1048576000
throttle带宽,单位是B,上面表示100M/s带宽。
3.验证迁移结果
./kafka-reassign-partitions.sh --zookeeper kafka001:2181/kafka --reassignment-json-file topic-todo.json --verify
分区多,数据量大的话,用时多一些,半个小时一个小时也是可能的。