Flume和Kafka的整合完成实时数据采集
来源:互联网 发布:金星舞蹈水平 知乎 编辑:程序博客网 时间:2024/05/19 21:40
1.Flume和Kafka整合框架
2.写FLUME的agent配置文件
exec-memory-avro.conf
exec-memory-avro.sources = exec-source
exec-memory-avro.sinks = avro-sink
exec-memory-avro.channels = memory-channel
exec-memory-avro.sources.exec-source.type = exec
exec-memory-avro.sources.exec-source.command = tail -F /home/bigdata/data/data.log
exec-memory-avro.sources.exec-source.shell = /bin/sh -c
exec-memory-avro.sinks.avro-sink.type = avro
exec-memory-avro.sinks.avro-sink.hostname = bigdata
exec-memory-avro.sinks.avro-sink.port = 44444
exec-memory-avro.channels.memory-channel.type = memory
exec-memory-avro.sources.exec-source.channels = memory-channel
exec-memory-avro.sinks.avro-sink.channel = memory-channel
avro-memory-kafka.conf
avro-memory-kafka.sources = avro-source
avro-memory-kafka.sinks = kafka-sink
avro-memory-kafka.channels = memory-channel
avro-memory-kafka.sources.avro-source.type = avro
avro-memory-kafka.sources.avro-source.bind = hadoop000
avro-memory-kafka.sources.avro-source.port = 44444
avro-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSink
avro-memory-kafka.sinks.kafka-sink.brokerList = hadoop000:9092
avro-memory-kafka.sinks.kafka-sink.topic = hello_topic
avro-memory-kafka.sinks.kafka-sink.batchSize = 5
avro-memory-kafka.sinks.kafka-sink.requiredAcks =1
avro-memory-kafka.channels.memory-channel.type = memory
avro-memory-kafka.sources.avro-source.channels = memory-channel
avro-memory-kafka.sinks.kafka-sink.channel = memory-channel
3.启动框架
(1)启动zookeeper
(2)启动kafka
(3)先启动FLUME的avro-memory-kafka.conf,再启动exec-memory-avro.conf
(4)启动kafkaconsumer
4.测试
echo hello world >>data.log
在consumer控制台上输出,搭建成功
- Flume和Kafka的整合完成实时数据采集
- Flume和Kafka完成实时数据的采集
- flume-kafka整合--实时日志采集
- Flume与Kafka整合完成实时数据处理
- flume采集数据到kafka和hive
- Flume实时采集数据
- kafka和flume整合
- Flume和Kafka整合
- [日志处理工作之一]整合elasticsearch,kibana,flume-ng,kafka实时采集tomcat日志
- flume实现kafka到hdfs实时数据采集 - 有负载均衡策略
- flume+kafka+zookeeper 单机实现实时数据的获取
- flume-1.4.0整合hbase-0.98.0实战数据采集,同时将采集到的数据放入hbase和h
- flume整合kafka和hdfs
- Flume和Kafka整合安装
- flume + Kafka采集数据 超简单
- flume与kafka的整合
- flume整合kafka之kafka接收flume数据
- flume抓取数据到kafka(整合)
- 自顶向下,逐步求精
- 51NOD 1004 n^n的末位数字
- 2步解决EF生成实体类无注释问题的快捷方法,亲测成功
- gulp-小练
- select函数与stdio混用的不良后果
- Flume和Kafka的整合完成实时数据采集
- ArrayList至SharedPreferences
- dede:if DEDE模板中使用php和if判断语句的技巧
- Springboot实现quartz定时
- NKOJ 4345 (Ipsc2015)Generating Synergy (DFS序+kd树)
- 嵌入式软件工程师经典面试题
- yii2的url
- POJ 2635.The Embarrassed Cryptographer
- python学习_6