Flume前述(三)--多 agent 汇聚写入 HDFS
来源:互联网 发布:super在java中的位置 编辑:程序博客网 时间:2024/06/05 04:09
客户机配置 Flume Client
# clientAgent
clientAgent.channels = c1
clientAgent.sources = s1
clientAgent.sinks = k1 k2
# clientAgent sinks group
clientAgent.sinkgroups = g1
# clientAgent Spooling Directory Source
clientAgent.sources.s1.type = spooldir
clientAgent.sources.s1.spoolDir =/home/lkm/logdata/
clientAgent.sources.s1.fileHeader = true
clientAgent.sources.s1.deletePolicy =immediate
clientAgent.sources.s1.batchSize =1000
clientAgent.sources.s1.channels =c1
clientAgent.sources.s1.deserializer.maxLineLength =1048576
# clientAgent FileChannel
clientAgent.channels.c1.type = file
clientAgent.channels.c1.checkpointDir = /flume/spool/checkpoint
clientAgent.channels.c1.dataDirs = /flume/spool/data
clientAgent.channels.c1.capacity = 200000000
clientAgent.channels.c1.keep-alive = 30
clientAgent.channels.c1.write-timeout = 30
clientAgent.channels.c1.checkpoint-timeout=600
# clientAgent Sinks
# k1 sink
clientAgent.sinks.k1.channel = c1
clientAgent.sinks.k1.type = avro
# connect to serverAgent
clientAgent.sinks.k1.hostname = r09n10
clientAgent.sinks.k1.port = 41415
# k2 sink
clientAgent.sinks.k2.channel = c1
clientAgent.sinks.k2.type = avro
# connect to CollectorBackupAgent
clientAgent.sinks.k2.hostname = r09n09
clientAgent.sinks.k2.port = 41415
# clientAgent sinks group
clientAgent.sinkgroups.g1.sinks = k1 k2
# load_balance type
clientAgent.sinkgroups.g1.processor.type = load_balance
clientAgent.sinkgroups.g1.processor.backoff = true
#clientAgent.sinkgroups.g1.processor.selector = random
clientAgent.sinkgroups.g1.processor.priority.k1 = 10
clientAgent.sinkgroups.g1.processor.priority.k2 = 9
执行命令:
flume-ng agent -n clientAgent -f client.conf -Dflume.root.logger=INFO,console
服务端配置Flume Server
# serverAgent
serverAgent.channels = c2
serverAgent.sources = s2
serverAgent.sinks =k1 k2
# serverAgent AvroSource
serverAgent.sources.s2.type = avro
serverAgent.sources.s2.bind = r09n10
serverAgent.sources.s2.port = 41415
serverAgent.sources.s2.channels = c2
# serverAgent FileChannel
serverAgent.channels.c2.type = file
serverAgent.channels.c2.checkpointDir =/flume/spool/checkpoint
serverAgent.channels.c2.dataDirs = /flume/spool/data,/flume/work/spool/data
serverAgent.channels.c2.capacity = 200000000
serverAgent.channels.c2.transactionCapacity=6000
serverAgent.channels.c2.checkpointInterval=60000
# serverAgent hdfsSink
serverAgent.sinks.k2.type = hdfs
serverAgent.sinks.k2.channel = c2
serverAgent.sinks.k2.hdfs.path = hdfs://nameservice1/flume/flume1
serverAgent.sinks.k2.hdfs.filePrefix =k2_%{file}
serverAgent.sinks.k2.hdfs.inUsePrefix =_
serverAgent.sinks.k2.hdfs.inUseSuffix =.tmp
serverAgent.sinks.k2.hdfs.rollSize = 0
serverAgent.sinks.k2.hdfs.rollCount = 0
serverAgent.sinks.k2.hdfs.rollInterval = 240
serverAgent.sinks.k2.hdfs.writeFormat = Text
serverAgent.sinks.k2.hdfs.fileType = DataStream
serverAgent.sinks.k2.hdfs.batchSize = 6000
serverAgent.sinks.k2.hdfs.callTimeout = 60000
serverAgent.sinks.k1.type = hdfs
serverAgent.sinks.k1.channel = c2
serverAgent.sinks.k1.hdfs.path = hdfs://nameservice1/flume/flume2
serverAgent.sinks.k1.hdfs.filePrefix =k1_%{file}
serverAgent.sinks.k1.hdfs.inUsePrefix =_
serverAgent.sinks.k1.hdfs.inUseSuffix =.tmp
serverAgent.sinks.k1.hdfs.rollSize = 0
serverAgent.sinks.k1.hdfs.rollCount = 0
serverAgent.sinks.k1.hdfs.rollInterval = 240
serverAgent.sinks.k1.hdfs.writeFormat = Text
serverAgent.sinks.k1.hdfs.fileType = DataStream
serverAgent.sinks.k1.hdfs.batchSize = 6000
serverAgent.sinks.k1.hdfs.callTimeout = 60000
执行命令:
flume-ng agent -n serverAgent -f server.conf -Dflume.root.logger=INFO,console
上面采用的就是类似 cs 架构,各个 flume agent 节点先将各台机器的日志汇总到 Consolidation 节点,然后再由这些节点统一写入 HDFS,并且采用了负载均衡的方式,你还可以配置高可用的模式等等。
- Flume前述(三)--多 agent 汇聚写入 HDFS
- flume学习(三):flume将log4j日志数据写入到hdfs
- flume学习(三):flume将log4j日志数据写入到hdfs
- Flume前述
- Flume前述(二)--功能配置
- flume学习(五):flume将log4j日志数据写入到hdfs
- flume学习(五):flume将log4j日志数据写入到hdfs
- flume学习(二):flume将log4j日志数据写入到hdfs
- Flume把网络数据写入HDFS
- flume 写入hdfs 采用lzo 格式 教程
- Hive读取Flume正在写入的HDFS
- flume的多agent连接
- Flume NG之Agent部署和sink配置HDFS
- flume流(三)如何用flume流从本地传输文件到hdfs?
- flume写入hadoop hdfs报错 Too many open files
- flume收集Hadoop运行产生的日志,并写入hdfs
- flume将log4j日志数据写入到hdfs
- hive读取与flume写入hdfs文件冲突
- Spring的事务管理阅读笔记与感悟(一)
- man命令详解
- Opencv自适应阀值操作-adaptiveThreshold函数
- 继承、接口小例子
- 作用域为类的常量
- Flume前述(三)--多 agent 汇聚写入 HDFS
- Java心得33
- ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’
- hdu4607 树的直径
- Opencv向上采样-pyrUp函数
- Linux命令全称
- Java心得34
- 数据库
- IO控制流的相关程序