kafka集群broker节点扩容方案
来源:互联网 发布:国内 阿里云 aws 优势 编辑:程序博客网 时间:2024/05/29 14:50
在用kafka集群有3个节点,即host1, host2, host3,现需要将broker节点扩容至6个,以提供更高的数据处理能力。
一、上架物理服务器用于提供更多的资源
新扩容3个broker节点,host4,host5,host6
二、在三台新增节点上部署kafka应用程序
这些不是本文的重点,略过。
三、重新分布原有的topic分区
1、查看集群中当前所有可用的topic
./kafka-topics.sh --list --zookeeper ip:port
- lcf-201612201649
- test-for-sys-monitor
2、查看特定topic的详细信息
./kafka-topics.sh --describe --zookeeper 192.168.1.92:2181 --topic lcf-201612201649Topic:lcf-201612201649 PartitionCount:24 ReplicationFactor:3 Configs: Topic: lcf-201612201649 Partition: 0 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2 Topic: lcf-201612201649 Partition: 1 Leader: 2 Replicas: 2,1,3 Isr: 3,1,2 Topic: lcf-201612201649 Partition: 2 Leader: 3 Replicas: 3,2,1 Isr: 3,1,2 Topic: lcf-201612201649 Partition: 3 Leader: 1 Replicas: 1,2,3 Isr: 1,3,2 Topic: lcf-201612201649 Partition: 4 Leader: 2 Replicas: 2,3,1 Isr: 3,1,2 Topic: lcf-201612201649 Partition: 5 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2 Topic: lcf-201612201649 Partition: 6 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2 Topic: lcf-201612201649 Partition: 7 Leader: 2 Replicas: 2,1,3 Isr: 1,3,2 Topic: lcf-201612201649 Partition: 8 Leader: 3 Replicas: 3,2,1 Isr: 3,1,2 Topic: lcf-201612201649 Partition: 9 Leader: 1 Replicas: 1,2,3 Isr: 1,3,2 Topic: lcf-201612201649 Partition: 10 Leader: 2 Replicas: 2,3,1 Isr: 3,1,2 Topic: lcf-201612201649 Partition: 11 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2 Topic: lcf-201612201649 Partition: 12 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2 Topic: lcf-201612201649 Partition: 13 Leader: 2 Replicas: 2,1,3 Isr: 3,1,2 Topic: lcf-201612201649 Partition: 14 Leader: 3 Replicas: 3,2,1 Isr: 3,1,2 Topic: lcf-201612201649 Partition: 15 Leader: 1 Replicas: 1,2,3 Isr: 1,3,2 Topic: lcf-201612201649 Partition: 16 Leader: 2 Replicas: 2,3,1 Isr: 3,1,2 Topic: lcf-201612201649 Partition: 17 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2 Topic: lcf-201612201649 Partition: 18 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2 Topic: lcf-201612201649 Partition: 19 Leader: 2 Replicas: 2,1,3 Isr: 1,3,2 Topic: lcf-201612201649 Partition: 20 Leader: 3 Replicas: 3,2,1 Isr: 3,1,2 Topic: lcf-201612201649 Partition: 21 Leader: 1 Replicas: 1,2,3 Isr: 1,3,2 Topic: lcf-201612201649 Partition: 22 Leader: 2 Replicas: 2,3,1 Isr: 3,1,2 Topic: lcf-201612201649 Partition: 23 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2
3、kafka集群分区日志迁移
我们的目标是把原来分布在3个节点上的topic(24partitions, 3replicas),将全部partitions重新分布到全部的6个节点上去。
(1) 先制作topics-to-move.json文件
cat << EOF > topic-to-move.json{"topics": [{"topic": "lcf-201612201649"}],"version":1}EOF
(2)使用-generate生成迁移计划
[testuser@c4 bin]$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.1.92:2181 --topics-to-move-json-file ./plans/topic-to-move.json --broker-list "1,2,3,4,5,6" --generateCurrent partition replica assignment{"version":1,"partitions":[{"topic":"lcf-201612201649","partition":1,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":8,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":19,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":15,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":18,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":13,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":0,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":10,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":5,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":12,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":17,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":9,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":7,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":20,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":23,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":3,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":2,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":4,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":11,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":6,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":14,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":22,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":16,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":21,"replicas":[1,2,3]}]}Proposed partition reassignment configuration{"version":1,"partitions":[{"topic":"lcf-201612201649","partition":1,"replicas":[4,1,2]},{"topic":"lcf-201612201649","partition":15,"replicas":[6,5,1]},{"topic":"lcf-201612201649","partition":8,"replicas":[5,3,4]},{"topic":"lcf-201612201649","partition":19,"replicas":[4,5,6]},{"topic":"lcf-201612201649","partition":13,"replicas":[4,3,5]},{"topic":"lcf-201612201649","partition":18,"replicas":[3,4,5]},{"topic":"lcf-201612201649","partition":0,"replicas":[3,6,1]},{"topic":"lcf-201612201649","partition":10,"replicas":[1,5,6]},{"topic":"lcf-201612201649","partition":5,"replicas":[2,5,6]},{"topic":"lcf-201612201649","partition":12,"replicas":[3,2,4]},{"topic":"lcf-201612201649","partition":9,"replicas":[6,4,5]},{"topic":"lcf-201612201649","partition":17,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":7,"replicas":[4,2,3]},{"topic":"lcf-201612201649","partition":20,"replicas":[5,6,1]},{"topic":"lcf-201612201649","partition":23,"replicas":[2,3,4]},{"topic":"lcf-201612201649","partition":3,"replicas":[6,3,4]},{"topic":"lcf-201612201649","partition":4,"replicas":[1,4,5]},{"topic":"lcf-201612201649","partition":2,"replicas":[5,2,3]},{"topic":"lcf-201612201649","partition":11,"replicas":[2,6,1]},{"topic":"lcf-201612201649","partition":6,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":22,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":14,"replicas":[5,4,6]},{"topic":"lcf-201612201649","partition":16,"replicas":[1,6,2]},{"topic":"lcf-201612201649","partition":21,"replicas":[6,1,2]}]}[testuser@c4 bin]$
注:生成一个向broker 1,2,3,4,5,6迁移指定topic的迁移计划。输出内容中包括当前的分布配置和即将改变后的分布配置。
将以上命令的输出内容保存为json文件。其中当前分布配置备份为backup.json,改变后的分布配置保存为expand-cluster-reassignment.json
(3)使用-execute执行迁移计划
[testuser@c4 kafka]$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.1.92:2181 --reassignment-json-file ./plans/reassignment-lcf-201612201649.json --executeCurrent partition replica assignment{"version":1,"partitions":[{"topic":"lcf-201612201649","partition":1,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":8,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":19,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":15,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":18,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":13,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":0,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":10,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":5,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":12,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":17,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":9,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":7,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":20,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":23,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":3,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":2,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":4,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":11,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":6,"replicas":[1,3,2]},{"topic":"lcf-201612201649","partition":14,"replicas":[3,2,1]},{"topic":"lcf-201612201649","partition":22,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":16,"replicas":[2,3,1]},{"topic":"lcf-201612201649","partition":21,"replicas":[1,2,3]}]}Save this to use as the --reassignment-json-file option during rollbackSuccessfully started reassignment of partitions {"version":1,"partitions":[{"topic":"lcf-201612201649","partition":12,"replicas":[3,2,4]},{"topic":"lcf-201612201649","partition":17,"replicas":[2,1,3]},{"topic":"lcf-201612201649","partition":8,"replicas":[5,3,4]},{"topic":"lcf-201612201649","partition":7,"replicas":[4,2,3]},{"topic":"lcf-201612201649","partition":23,"replicas":[2,3,4]},{"topic":"lcf-201612201649","partition":4,"replicas":[1,4,5]},{"topic":"lcf-201612201649","partition":19,"replicas":[4,5,6]},{"topic":"lcf-201612201649","partition":2,"replicas":[5,2,3]},{"topic":"lcf-201612201649","partition":20,"replicas":[5,6,1]},{"topic":"lcf-201612201649","partition":11,"replicas":[2,6,1]},{"topic":"lcf-201612201649","partition":16,"replicas":[1,6,2]},{"topic":"lcf-201612201649","partition":5,"replicas":[2,5,6]},{"topic":"lcf-201612201649","partition":14,"replicas":[5,4,6]},{"topic":"lcf-201612201649","partition":10,"replicas":[1,5,6]},{"topic":"lcf-201612201649","partition":18,"replicas":[3,4,5]},{"topic":"lcf-201612201649","partition":22,"replicas":[1,2,3]},{"topic":"lcf-201612201649","partition":15,"replicas":[6,5,1]},{"topic":"lcf-201612201649","partition":1,"replicas":[4,1,2]},{"topic":"lcf-201612201649","partition":6,"replicas":[3,1,2]},{"topic":"lcf-201612201649","partition":21,"replicas":[6,1,2]},{"topic":"lcf-201612201649","partition":13,"replicas":[4,3,5]},{"topic":"lcf-201612201649","partition":0,"replicas":[3,6,1]},{"topic":"lcf-201612201649","partition":9,"replicas":[6,4,5]},{"topic":"lcf-201612201649","partition":3,"replicas":[6,3,4]}]}
(4)使用--verify进行迁移结果的验证
[testuser@c4 kafka]$ ./bin/kafka-reassign-partitions.sh --zookeeper 192.168.1.92:2181 --reassignment-json-file ./plans/reassignment-lcf-201612201649.json --verify
查看重分布结果:
./bin/kafka-topics.sh --describe --zookeeper 192.168.1.92:2181 --topic lcf-201612201649
当以上迁移过程导致kafka的leader分布,不符preferred replica分布建议,则可以手动进行再平衡维护。
注:进行分区迁移时,最好先保留一个分区在原来的磁盘,这样不会影响正常的消费和生产。部分迁移则支持正常消费和生产。
(5)关于kafka集群leader平衡机制的维护
./kafka-preferred-replica-election.sh --zookeeper ip:port
或者在配置文件中将参数设置为:
auto.leader.rebalance.enable=true
阅读全文
0 0
- kafka集群broker节点扩容方案
- Kafka 单节点单Kafka Broker集群
- Kafka 单节点多Kafka Broker集群
- kafka集群下线broker节点实践方法
- kafka broker集群
- Kafka 集群的扩容
- kafka集群扩容(Topic迁移)
- Kafka学习笔记:设置多broker集群
- Kafka学习整理九(集群的扩容)
- Kafka学习整理九(集群的扩容)
- kafka集群扩容以及数据迁移
- kafka环境搭建2-broker集群+zookeeper集群
- 各节点集群方案
- zookeeper集群扩容/下线节点实践
- kafka集群3节点搭建
- kafka集群/单节点部署
- kafka+zk 集群--3个broker 3个zookeeper
- kafka 集群--3个broker 3个zookeeper创建实战
- java--继承(2)
- java脚本
- Jzoj1306 Sum
- hdu 5546 Ancient Go(bfs+模拟)
- Java 静态分派与动态分派
- kafka集群broker节点扩容方案
- HDU 6208
- clipse–>run as –> maven test 出现中文乱码
- Cisco Packet Tracer使用方法和路由器基本配置
- python字符串函数
- Android:控件WebView显示网页
- Legal or Not(拓扑排序)
- bzoj 1694 && 1742: [Usaco2005 nov]Grazing on the Run 边跑边吃草(DP)
- Git入门介绍-1-简单介绍