flume与kafka的整合
来源:互联网 发布:淘宝小视频链接地址 编辑:程序博客网 时间:2024/06/05 09:48
flume,kafka 是基于zookeeper的,前安装zookeeper
1 搭建kafka集群 详见
注意:1各个节点的broker id 要进行分配 ,不能相同,建议 按顺序排列 0,1,2(我是三个节点)
2 hostname = node1 port=9092
3 zookeeper connect = node1:2181,node2:2181,node3:2181
2 启动各个节点上的zookeeper
zkServer.sh start ------>zkServer.sh status(查看状态)
3 启动各个节点上 kafka
进入根目录 ./bin/kafka-server-start.sh config/server.properties
4 创建Topic 和分配partition
./bin/kafka-topics.sh -zookeeper node1:2181,node2:2181,node3:2181 -topic topicTest -replication-factor 2 -partitions 5 --create
5 创建生产者(主节点)
./bin/kafka-console-producer.sh --broker-list node1:9092,node2:9092,node3:9092 --topic testTopic
查看Topic
./bin/kafka-topics.sh -zookeeper node1:2181,node2:2181,node3:2181 -list
6 创建消费者(各个节点)
./bin/kafka-console-consumer.sh --zookeeper node1:2181,node2:2181,node3:2181 --from-beginning -topic testTopic
7 验证消费能否接受到消息
在producer 窗口内写入数据,consumer端是否可以接受数据
附:kafka配置文件
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
# The port the socket server listens on
port=9092
# Hostname the broker will bind to. If not set, the server will bind to all interfaces
host.name=node1
# A comma seperated list of directories under which to store log files
log.dirs=/tmp/kafka-logs(查看写入文件日志)
zookeeper.connect=node1:2181,node2:2181,node3:2181
只以上做了修改
搭建flume
我选取的是单节点监控问价目录
1 flume安装详见flume搭建
2 配置文件
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = spooldir
a1.sources.r1.spoolDir = /opt/data/flumedata
a1.sources.r1.fileHeader = true
a1.sources.r1.interceptors = i1
a1.sources.r1.deserializer.maxLineLength = 102400
a1.sources.r1.interceptors.i1.type = timestamp
# Describe the sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic = testTopic
a1.sinks.k1.brokerList = node1:9092,node2:9092,node3:9092
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 10000
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
3 启动flume服务
./bin/flume-ng agent -c ./conf/ -f conf/flume-conf.properties -Dflume.root.logger=DEBUG,console -na1
4 向文件中写入数据
cp /opt/soft/liuliqiao/* /opt/data/flumedata/
5 同时查看consumer窗口
有数据接收
同时可以在日志文件查看
log.dirs=/tmp/kafka-logs
整合ok
- flume与kafka的整合
- Flume与Kafka整合
- Flume与Kafka整合
- Flume与Kafka整合
- flume与kafka整合
- Flume,Kafka与Storm整合
- flume-ng与kafka整合
- Flume与Kafka整合案例详解
- Flume与Kafka整合案例详解
- Flume与Kafka整合完成实时数据处理
- Flume与Kafka整合案例详解
- Flume+kafka+storm整合
- Kafka flume 整合
- Kafka flume 整合
- Flume+Kafka+SparkStreaming整合
- flume+kafka+storm整合
- 103-flume整合kafka
- Flume+Kafka整合
- 神经网络与深度学习(1)
- Java Spring搭配Junit测试配置
- 机器学习(8)-支持向量机
- 第十三篇:JAVA对象持久化
- 博客专家申请规则变更调整
- flume与kafka的整合
- C#207课的主要内容
- 文章标题
- 各种加解密算法比较
- Win10把文件刻录成数据光盘
- AES加密和解密
- codility Fish
- Object类作用
- VS Code 折腾记