Flume-ng分布式部署和配置
来源:互联网 发布:机动车扣分查询软件 编辑:程序博客网 时间:2024/05/21 14:45
Flume-ng分布式部署总结
agent.channels = memoryChannel
agent.sinks = hdfsSink
# For each one of the sources, the type is defined
agent.sources.avrosrc.type = avro
agent.sources.avrosrc.bind = 192.168.35.100
agent.sources.avrosrc.port = 44444
# The channel can be defined as follows.
agent.sources.avrosrc.channels = memoryChannel
# Each channel's type is defined.
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.keep-alive = 10
agent.channels.memoryChannel.capacity = 100000
agent.channels.memoryChannel.transactionCapacity =100000
# Each sink's type must be defined
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.channel = memoryChannel
agent.sinks.hdfsSink.hdfs.rollInterval = 0
agent.sinks.hdfsSink.hdfs.rollSize = 4000000
agent.sinks.hdfsSink.hdfs.rollCount = 0
agent.sinks.hdfsSink.hdfs.writeFormat = Text
agent.sinks.hdfsSink.hdfs.fileType = DataStream
agent.sinks.hdfsSink.hdfs.batchSize = 10
2.配置收集日志各节点的Flume的配置文件,flume的安装目录下/conf/flume-conf.properties
agent.sources = nodesource
agent.channels = nodeMemoryChannel
agent.sinks = nodeSink
agent.sources.nodesource.type = exec
agent.sources.nodesource.command = tail -F /root/logs/log1.log #监听的日志文件
agent.sources.nodesource.channels = nodeMemoryChannel
agent.sources.nodesource.interceptors = host_int timestamp_int inter1
agent.sources.nodesource.interceptors.host_int.type = host
agent.sources.nodesource.interceptors.host_int.hostHeader = hostname
agent.sources.nodesource.interceptors.timestamp_int.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
#agent.sources.nodesource.interceptors = inter1
agent.sources.nodesource.interceptors.inter1.value = log102 #在数据接收端生成的文件名称
agent.channels.nodeMemoryChannel.type = memory
agent.channels.nodeMemoryChannel.keep-alive = 10
agent.channels.nodeMemoryChannel.capacity = 100000
agent.channels.nodeMemoryChannel.transactionCapacity =100000
agent.sinks.nodeSink.type = avro
agent.sinks.nodeSink.hostname = 192.168.35.100
agent.sinks.nodeSink.port = 44444
agent.sinks.nodeSink.channel = nodeMemoryChannel
3.启动(切换到flume的安装路径下启动)
#将日志信息打印到控制台
bin/flume-ng agent -n agent -c conf -f conf/flume-conf.properties -Dflume.root.logger=INFO,console
#将产生的日志信息保存到日志文件中,并后台运行
bin/flume-ng agent -n agent -c conf -f conf/flume-conf.properties >logs/flume_log.log &
1.日志总接收端的配置(负责接收各个节点发送过来的日志数据),修改flume的安装目录下/conf/flume-conf.properties配置文件
(默认没有,cp flume-conf.properties.template flume-conf.properties)
agent.sources = avrosrcagent.channels = memoryChannel
agent.sinks = hdfsSink
# For each one of the sources, the type is defined
agent.sources.avrosrc.type = avro
agent.sources.avrosrc.bind = 192.168.35.100
agent.sources.avrosrc.port = 44444
# The channel can be defined as follows.
agent.sources.avrosrc.channels = memoryChannel
# Each channel's type is defined.
agent.channels.memoryChannel.type = memory
agent.channels.memoryChannel.keep-alive = 10
agent.channels.memoryChannel.capacity = 100000
agent.channels.memoryChannel.transactionCapacity =100000
# Each sink's type must be defined
agent.sinks.hdfsSink.type = hdfs
agent.sinks.hdfsSink.channel = memoryChannel
agent.sinks.hdfsSink.hdfs.path = /flume_logs/%Y%m%d #存放文件的目录结构
#生成的文件的名称,“datacenter”各节点配置文件中定义的名称
agent.sinks.hdfsSink.hdfs.rollInterval = 0
agent.sinks.hdfsSink.hdfs.rollSize = 4000000
agent.sinks.hdfsSink.hdfs.rollCount = 0
agent.sinks.hdfsSink.hdfs.writeFormat = Text
agent.sinks.hdfsSink.hdfs.fileType = DataStream
agent.sinks.hdfsSink.hdfs.batchSize = 10
2.配置收集日志各节点的Flume的配置文件,flume的安装目录下/conf/flume-conf.properties
agent.sources = nodesource
agent.channels = nodeMemoryChannel
agent.sinks = nodeSink
agent.sources.nodesource.type = exec
agent.sources.nodesource.command = tail -F /root/logs/log1.log #监听的日志文件
agent.sources.nodesource.channels = nodeMemoryChannel
agent.sources.nodesource.interceptors = host_int timestamp_int inter1
agent.sources.nodesource.interceptors.host_int.type = host
agent.sources.nodesource.interceptors.host_int.hostHeader = hostname
agent.sources.nodesource.interceptors.timestamp_int.type = org.apache.flume.interceptor.TimestampInterceptor$Builder
#agent.sources.nodesource.interceptors = inter1
agent.sources.nodesource.interceptors.inter1.type = static
#数据接收端通过“datacenter”获取value值定义文件名称
agent.sources.nodesource.interceptors.inter1.value = log102 #在数据接收端生成的文件名称
agent.channels.nodeMemoryChannel.type = memory
agent.channels.nodeMemoryChannel.keep-alive = 10
agent.channels.nodeMemoryChannel.capacity = 100000
agent.channels.nodeMemoryChannel.transactionCapacity =100000
agent.sinks.nodeSink.type = avro
agent.sinks.nodeSink.hostname = 192.168.35.100
agent.sinks.nodeSink.port = 44444
agent.sinks.nodeSink.channel = nodeMemoryChannel
3.启动(切换到flume的安装路径下启动)
#将日志信息打印到控制台
bin/flume-ng agent -n agent -c conf -f conf/flume-conf.properties -Dflume.root.logger=INFO,console
#将产生的日志信息保存到日志文件中,并后台运行
bin/flume-ng agent -n agent -c conf -f conf/flume-conf.properties >logs/flume_log.log &
0 0
- Flume-ng分布式部署和配置
- Flume NG之Agent部署和sink配置HDFS
- Flume NG 部署实践
- flume-ng和kafka集成的配置
- flume伪分布式部署和测试
- Flume NG 配置详解
- Flume-ng配置
- Flume NG 配置
- Flume NG 配置详解
- Flume-ng配置
- flume ng高可用部署
- Flume NG之Agent部署和sink配置HDFS且吐槽CSDN博客及客服态度
- Flume NG 学习笔记(五)Sinks和Channel配置
- flume ng配置拓扑图
- flume ng配置拓扑图
- Flume-ng agent配置说明
- Flume-ng 高级功能配置
- Flume 1.4.0 NG 分布式集群搭建
- JUnit白盒测试-第3天
- 动态规则装配线调度
- VC++ warning C4786 详解
- android widget使用
- 认识Javascript中的navigator对象以及检测浏览器类型和版本,获取浏览器版本号,检测客户端操作系统
- Flume-ng分布式部署和配置
- 冯东阳:吐胆汁分享原创文章写作经验
- CentOS(Linux) VPS 防暴力登录账户和密码
- Jquery 怎么只给点击的元素加个class(互斥)
- HDU 1599 find the mincost route 无向图最小环
- 错误 18456 ___ODBC 创建新数据源
- nyoj-984-最长回文
- javascript:void(0)链接
- [Google Guava] 2.1-不可变集合