读取Kafka集群的消息
来源:互联网 发布:中国经济增长速度 知乎 编辑:程序博客网 时间:2024/04/28 20:56
读取Kafka集群的消息
Table of Contents
- 1. 先启动zookeeper
- 2. 本地启动多个brokers
- 3. producer产生多个消息,分布在一个topic的2个partition中
- 3.1. 创建一个test2 topic
- 3.2. 查看下topic信息
- 3.3. 写入消息
- 4. consumer读取所有brokers的指定topic的所有消息
1 先启动zookeeper
kafka安装包自带了zookeeper程序,很方便。 进入bin目录,运行下面的脚本
[dean@dell_xps_13 bin]$ ./zookeeper-server-start.sh ../config/zookeeper.properties [2016-04-29 16:03:41,484] INFO Reading configuration from: ../config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)[2016-04-29 16:03:41,484] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)[2016-04-29 16:03:41,498] INFO Reading configuration from: ../config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)[2016-04-29 16:03:41,499] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)[2016-04-29 16:03:41,503] INFO Server environment:zookeeper.version=3.3.3-1203054, built on 11/17/2011 05:47 GMT (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,503] INFO Server environment:host.name=localhost.localdomain (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,503] INFO Server environment:java.version=1.8.0_77 (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,503] INFO Server environment:java.vendor=Oracle Corporation (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,503] INFO Server environment:java.home=/usr/lib/jvm/java-8-openjdk/jre (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,503] INFO Server environment:java.class.path=:/home/dean/work/software/kafka_2.10-0.8.1/bin/../core/build/dependant-libs-2.8.0/*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../perf/build/libs//kafka-perf_2.8.0*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../clients/build/libs//kafka-clients*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../examples/build/libs//kafka-examples*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/jopt-simple-3.2.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/kafka_2.10-0.8.1.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/log4j-1.2.15.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/metrics-annotation-2.2.0.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/metrics-core-2.2.0.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/scala-library-2.10.1.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/slf4j-api-1.7.2.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/snappy-java-1.0.5.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/zkclient-0.3.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../libs/zookeeper-3.3.4.jar:/home/dean/work/software/kafka_2.10-0.8.1/bin/../core/build/libs/kafka_2.8.0*.jar (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,503] INFO Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,503] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,503] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,503] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,503] INFO Server environment:os.arch=amd64 (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,503] INFO Server environment:os.version=4.5.1-1-ARCH (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,503] INFO Server environment:user.name=dean (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,503] INFO Server environment:user.home=/home/dean (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,503] INFO Server environment:user.dir=/home/dean/work/software/kafka_2.10-0.8.1/bin (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,507] INFO tickTime set to 3000 (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,507] INFO minSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,507] INFO maxSessionTimeout set to -1 (org.apache.zookeeper.server.ZooKeeperServer)[2016-04-29 16:03:41,520] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxn)[2016-04-29 16:03:41,535] INFO Snapshotting: 0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)[2016-04-29 16:10:44,072] INFO Accepted socket connection from /0:0:0:0:0:0:0:1:34216 (org.apache.zookeeper.server.NIOServerCnxn)[2016-04-29 16:10:44,111] INFO Client attempting to establish new session at /0:0:0:0:0:0:0:1:34216 (org.apache.zookeeper.server.NIOServerCnxn)[2016-04-29 16:10:44,113] INFO Creating new log file: log.1 (org.apache.zookeeper.server.persistence.FileTxnLog)[2016-04-29 16:10:44,118] INFO Established session 0x154610b75630000 with negotiated timeout 30000 for client /0:0:0:0:0:0:0:1:34216 (org.apache.zookeeper.server.NIOServerCnxn)[2016-04-29 16:24:37,145] INFO Processed session termination for sessionid: 0x154610b75630000 (org.apache.zookeeper.server.PrepRequestProcessor)[2016-04-29 16:24:37,151] INFO Closed socket connection for client /0:0:0:0:0:0:0:1:34216 which had sessionid 0x154610b75630000 (org.apache.zookeeper.server.NIOServerCnxn)
2 本地启动多个brokers
这里打算启动两个broker, 需要两个配置文件,进入config目录后复制出两个文件
cp server.properties server1.propertiescp server.properties server2.properties
两个配置文件里面分别修改3处: server1.properties
############################# Server Basics ############################## The id of the broker. This must be set to a unique integer for each broker.broker.id=1############################# Socket Server Settings ############################## The port the socket server listens onport=9093############################# Log Basics ############################## A comma seperated list of directories under which to store log fileslog.dirs=/tmp/kafka-logs-1
server2.properties
############################# Server Basics ############################## The id of the broker. This must be set to a unique integer for each broker.broker.id=2############################# Socket Server Settings ############################## The port the socket server listens onport=9094############################# Log Basics ############################## A comma seperated list of directories under which to store log fileslog.dirs=/tmp/kafka-logs-2
运行命令分别启动两个brokers
./kafka-server-start.sh ../config/server1.properties ./kafka-server-start.sh ../config/server2.properties
3 producer产生多个消息,分布在一个topic的2个partition中
3.1 创建一个test2 topic
./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 2 --topic test2Created topic "test2".
3.2 查看下topic信息
./kafka-topics.sh --describe --zookeeper localhost:2181 --topic test2Topic:test2 PartitionCount:2 ReplicationFactor:2 Configs: Topic: test2 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2 Topic: test2 Partition: 1 Leader: 2 Replicas: 2,1 Isr: 2,1
3.3 写入消息
用producer脚本写入多个消息
./kafka-console-producer.sh --broker-list localhost:9093,localhost:9094 --topic test2SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to no-operation (NOP) logger implementationSLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.shanghaibeijinghangzhoujinanyangzhoushenzhenghongkongnanjing
4 consumer读取所有brokers的指定topic的所有消息
用consumer脚本读取
./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test2 --from-beginningSLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".SLF4J: Defaulting to no-operation (NOP) logger implementationSLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.shanghaibeijinghangzhoujinanyangzhoushenzhenghongkongnanjing
读取成功。
Created: 2016-05-01 Sun 21:40
Validate
0 0
- 读取Kafka集群的消息
- kafka集群的步骤
- kafka集群的搭建
- Kafka 集群的扩容
- 分布式消息队列Kafka集群安装
- 分布式消息队列Kafka集群安装
- 分布式消息队列Kafka集群安装
- kafka入门1-集群生产消息
- 分布式消息队列Kafka集群安装
- Kafka消息处理与集群维护
- 动态从zookeeper中读取kafka集群
- redis集群和kafka集群作为消息队列比较
- storm读取kafka集群各个集群节点信息
- Spark Streaming +Kafka 使用底层API直接读取Kafka的Partition数据,手动更新Offset到Zookeeper集群
- 分布式的消息系统Kafka
- kafka消息队列的安装
- Kafka的消息可靠传递
- 集群外机器做producer往kafka里写消息
- JAVA语言之计数排序
- Geekband009第九周笔记分享
- 动态删除ArrayList中的元素
- 找到满足条件的数组
- Javascript事件全解析——事件概述及传统事件类型
- 读取Kafka集群的消息
- php环境搭建
- java热部署及类加载
- HTTP协议
- java_包
- 【转】ANDROID APK的数字签名的作用和意义
- 用户的退出登录功能
- UVa Self-Assembly(topo sort)[待AC]
- 二、String类中的判断