kafka集群扩容(Topic迁移)

来源:互联网 发布:网络团伙诈骗案判刑 编辑:程序博客网 时间:2024/06/16 03:53

文章新地址

文章开始前推荐使用 kafka-manager 工具,可以实时查看kafka同步状态,broker实时读写流量,topic分区情况等信息,提供重选最优leader等操作。

kafka的集群扩容实际上就是把 topic 的 partition 移动到新加的集群上。

生成 topic 移动 json文件有两种方式:

  1. 通过 --topics-to-move-json-file--broker-list 批量生成新的topic分区信息,然后根据该信息执行转移操作。

  2. 手动写要移动的topic信息,更灵活,但是在大量 topic 和 partition 的情况下非常繁琐并且容易出错。

1. 启动新节点

将原节点上的 kafka 目录通过 scp 命令拷贝到新节点,只需要修改配置文件中的 broker_id 和 ip 地址,然后依次启动 kafka 服务。

2. 移动topic

移动 topicA , topicB

sudo vim topics-to-move.json{"topics": [{"topic": "topicA"},            {"topic": "topicB"}],"version":1}

将上述 topic 移动到 broker 3,4,5上,用 generate 命令生成 partition 分配 json 串。

bin/kafka-reassign-partitions.sh --zookeeper zk1:2181 --topics-to-move-json-file topics-to-move.json --broker-list "3,4,5" --generate 

将看到如下:

Current partition replica assignment{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[0,2]},{"topic":"topicA","partition":1,"replicas":[1,0]},{"topic":"topicB","partition":1,"replicas":[0,1]},{"topic":"topicA","partition":0,"replicas":[2,1]},{"topic":"topicB","partition":0,"replicas":[2,0]},{"topic":"topicB","partition":2,"replicas":[1,2]}]}Proposed partition reassignment configuration{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[3,4]},{"topic":"topicA","partition":1,"replicas":[5,3]},{"topic":"topicB","partition":1,"replicas":[3,4]},{"topic":"topicA","partition":0,"replicas":[4,5]},{"topic":"topicB","partition":0,"replicas":[5,3]},{"topic":"topicB","partition":2,"replicas":[4,5]}]}

Proposed partition reassignment configuration 下方的数据写入 expand-cluster-reassignment.json

上文说的第二种方式其实就是手动写该文件,而不用命令生成的。

vim expand-cluster-reassignment.json{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[3,4]},{"topic":"topicA","partition":1,"replicas":[5,3]},{"topic":"topicB","partition":1,"replicas":[3,4]},{"topic":"topicA","partition":0,"replicas":[4,5]},{"topic":"topicB","partition":0,"replicas":[5,3]},{"topic":"topicB","partition":2,"replicas":[4,5]}]}

execute 命令开始正式执行,将会把上述2个topic按 expand-cluster-reassignment 中的设置进行分配,该过程可能会影响其它topic的读取(延时)

bin/kafka-reassign-partitions.sh --zookeeper zk1:2181 --reassignment-json-file expand-cluster-reassignment.json --execute

执行状态查询

查询执行状态

bin/kafka-reassign-partitions.sh --zookeeper zk1:2181 --reassignment-json-file expand-cluster-reassignment.json --verify

主要有几种状态

Reassignment of partition [topicA,4] is still in progress # 转移中Reassignment of partition [topicB,2] completed successfully # 转移结束

如果你看到一大堆的 ERROR 信息:

ERROR: Assigned replicas (0,5,1,2,3,4) don't match the list of replicas for reassignment (5,3,4) for partition [topicA,3]Reassignment of partition [topicA,3] failed

不要方,不要关闭节点,过会再看看就正常了。

1 0
原创粉丝点击