Kafka(二):环境搭建&测试
来源:互联网 发布:centos 安装desktop 编辑:程序博客网 时间:2024/05/01 11:12
本文基于Kafka 0.8
在一台机器上构建一个3个节点的kafka集群,并测试producer、consumer在正常情况下的行为,以及在lead broker/follow broker失效情况下的行为
1.下载并解压kafka 0.8.0 release
$ mkdir kafka
$ wget http://apache.dataguru.cn/kafka/0.8.0/kafka_2.8.0-0.8.0.tar.gz
$ tar -zxvf kafka_2.8.0-0.8.0.tar.gz
$ cd kafka_2.8.0-0.8.0
$ ll
total 2560
drwxr-xr-x 6 root root 4096 Dec 17 17:44 ./
drwxr-xr-x 4 root root 4096 Dec 17 18:20 ../
drwxr-xr-x 3 root root 4096 Dec 17 18:16 bin/
drwxr-xr-x 2 root root 4096 Dec 17 17:43 config/
-rw-r--r-- 1 root root 2520145 Nov 27 06:21 kafka_2.8.0-0.8.0.jar
drwxr-xr-x 2 root root 4096 Nov 27 06:21 libs/
-rw-r--r-- 1 root root 12932 Nov 27 06:21 LICENSE
drwxr-xr-x 2 root root 4096 Dec 17 18:00 logs/
-rw------- 1 root root 47165 Dec 17 18:10 nohup.out
-rw-r--r-- 1 root root 162 Nov 27 06:21 NOTICE
2.启动一个单节点的zookeeper
$ nohup bin/zookeeper-server-start.sh config/zookeeper.properties &
3. 准备启动一个3个broker节点的kafka集群,因此做如下配置
$ cp config/server.properties config/server-1.properties
$ cp config/server.properties config/server-2.properties
并做如下修改:
config/server-1.properties:
broker.id=1
port=9093
log.dir=/tmp/kafka-logs-1
config/server-2.properties:
broker.id=2
port=9094
log.dir=/tmp/kafka-logs-2
说明:
broker.id: broker节点的唯一标识
port: broker节点使用端口号
log.dir: 消息目录位置
4. 启动3个broker节点
$ JMX_PORT=9997 bin/kafka-server-start.sh config/server-1.properties &
$ JMX_PORT=9998 bin/kafka-server-start.sh config/server-2.properties &
$ JMX_PORT=9999 bin/kafka-server-start.sh config/server.properties &
5. 创建topic并查看
$ bin/kafka-create-topic.sh --zookeeper localhost:2181 --replica 3 --partition 1 --topic 3test
creation succeeded!
$ bin/kafka-list-topic.sh --zookeeper localhost:2181
topic: 3test partition: 0 leader: 2 replicas: 2,1,0 isr: 2,1,0
topic: test partition: 0 leader: 0 replicas: 0 isr: 0
topic: test_topic partition: 0 leader: 1 replicas: 0,1,2 isr: 1,2,0
说明:
partiton: partion id,由于此处只有一个partition,因此partition id 为0
leader:当前负责读写的lead broker id
relicas:当前partition的所有replication broker list
isr:relicas的子集,只包含出于活动状态的broker
6.启动consumer & producer,并在producer启动后的console输入一些信息
$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --from-beginning --topic 3test
message1
message3
message2
$ bin/kafka-console-producer.sh --broker-list localhost:9092,localhost:9093,localhost:9094 --topic 3test
message1
message3
message2
producer发送的数据consumer都能正常消费
7. 干掉follow broker
杀掉一个非lead broker(lead broker id为2)
$ pkill -9 -f server-1.properties
查看topic:
$ bin/kafka-list-topic.sh --zookeeper localhost:2181
topic: 3test partition: 0 leader: 2 replicas: 2,1,0 isr: 2,0
topic: test partition: 0 leader: 0 replicas: 0 isr: 0
topic: test_topic partition: 0 leader: 2 replicas: 0,1,2 isr: 2,0
此时,存活的broker只有2,0
测试:produce发送消息,consumer能正常接收到
8. 继续干掉leader broker
$ pkill -9 -f server-2.properties
$ bin/kafka-list-topic.sh --zookeeper localhost:2181
topic: 3test partition: 0 leader: 2 replicas: 2,1,0 isr: 2,0
topic: test partition: 0 leader: 0 replicas: 0 isr: 0
topic: test_topic partition: 0 leader: 2 replicas: 0,1,2 isr: 2,0
$ bin/kafka-list-topic.sh --zookeeper localhost:2181
topic: 3test partition: 0 leader: 2 replicas: 2,1,0 isr: 2,0
topic: test partition: 0 leader: 0 replicas: 0 isr: 0
topic: test_topic partition: 0 leader: 2 replicas: 0,1,2 isr: 2,0
$ bin/kafka-list-topic.sh --zookeeper localhost:2181
topic: 3test partition: 0 leader: 0 replicas: 2,1,0 isr: 0
topic: test partition: 0 leader: 0 replicas: 0 isr: 0
topic: test_topic partition: 0 leader: 0 replicas: 0,1,2 isr: 0
$ bin/kafka-list-topic.sh --zookeeper localhost:2181
topic: 3test partition: 0 leader: 0 replicas: 2,1,0 isr: 0
topic: test partition: 0 leader: 0 replicas: 0 isr: 0
topic: test_topic partition: 0 leader: 0 replicas: 0,1,2 isr: 0
杀掉leader broker过了一会,broker 0成为新的leader broker
测试:produce发送消息,consumer能正常接收到
参考:Kafka QuickStart
2 2
- Kafka(二):环境搭建&测试
- Kafka(二):环境搭建&测试
- Kafka分布式消息队列(二):环境搭建&测试
- kafka本地测试环境搭建
- kafka单机测试环境搭建
- kafka单机版环境搭建与测试
- kafka单机版环境搭建与测试
- kafka单机环境搭建与测试
- 02-kafka环境搭建及测试
- Kafka使用入门教程之二--搭建Kafka运行环境
- Apache Kafka系列(二)----Windwos下搭建Kafka环境
- 二. 搭建开发测试环境
- Kafka环境搭建及与Spring的整合测试
- Windows下Storm+Kafka+Zookeeper单机开发环境搭建测试
- kafka学习二:在linux环境下kafka搭建环境(单机版)
- kafka学习二:在linux环境下kafka搭建环境(单机版)
- kafka环境搭建二---Windows客户端Linux服务器
- Kafka开发实战(二)-集群环境搭建
- MATLAB Toolbox Path Cache is out of date and is not being used的解决
- copy命令玩捆绑
- gabor滤波器整理
- C++反汇编学习笔记1——选择判断语句
- 求字符串的长度(指针作参考)
- Kafka(二):环境搭建&测试
- Ubuntu系统较全面清理
- 移动开发者的自学宝典:十大在线编程学习网站
- vc下cstring转换为float
- Ubuntu 图形化安装
- shareSDK集成
- 跨浏览器插件框架FireBreath安装与使用之一——Windows下的插件
- 国人当自强:两岸三地在线编程学习网站大搜罗
- Ubuntu简单学命令轻松装软件