apache flume agent安装
来源:互联网 发布:算法第四版课后题答案 编辑:程序博客网 时间:2024/05/16 07:08
1、Flume Agent安装(采用spoolDir模式以便获取系统、应用等日志信息)
注:使用jyapp用户安装
一台虚拟机部署了多个JAVA应用同时需要部署多个flume-agent进行监控时,
需要调整如下配置文件:
a flume-agent/conf/app.conf中的spool_Dir参数
b flume-agent/conf/flume-env.sh中的jmxremote.port参数
c logback.xml中fileNamePattern参数
d flume_spooldir.sh中的dir参数
1.1 安装环境
系统:
CentOS release 6.6
软件:
flume-agent.tar.gz
flume-agent.tar.gz和flume_spooldir.sh文件下载路径:ftp://172.21.100.61/pro/flume/flume-agent
1.2 安装步骤
1.2.1 部署Flume Agent
具体脚本(jyapp用户):
cd /home/jyapp
mkdir flume_logs
tar -zxvf flume-agent.tar.gz
cd flume-agent
#修改flume环境变量
vi conf/flume-env.sh
具体配置解析:(注意:红色需修改)
------------------------------------------------------
export JAVA_OPTS="-Xms100m -Xmx100m -Dflume.root.logger=INFO,LOGFILE -Dflume.monitoring.type=http -Dflume.monitoring.port=34545 -Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=本机IP -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
-------------------------------------------------------------------------------------
#修改agent脚本
vi conf/app.conf
具体配置解析:(注意:红色需修改)
------------------------------------------------------
app.sources = r1
app.sinks = k1 k2
app.channels = c1
app.sources.r1.type = spooldir
app.sources.r1.spoolDir = /home/jyapp/flume_logs #定义监控的日志目录
app.sources.r1.deletePolicy = immediate
app.sources.r1.ignorePattern = ^(.)*\\.tmp$ #忽略.tmp文件
app.sources.r1.deserializer = MULTILINE #使用多行反序列化器
app.sources.r1.deserializer.maxLineLength = 60000
app.sources.r1.interceptors = i1 i2 i3
app.sources.r1.interceptors.i1.type = timestamp #设置时间戳
app.sources.r1.interceptors.i2.type = static
app.sources.r1.interceptors.i2.key = flume.client.ip
app.sources.r1.interceptors.i2.value = 客户机IP地址 #设置IP地址
app.sources.r1.interceptors.i3.type = static
app.sources.r1.interceptors.i3.key = flume.client.system.flag
app.sources.r1.interceptors.i3.value = 客户机系统标示 #设置系统标示,如S001
app.channels.c1.type = memory
app.channels.c1.capacity = 1000
app.channels.c1.transactionCapacity = 1000
app.channels.c1.keep-alive = 3
app.channels.c1.byteCapacityBufferPercentage = 30
app.channels.c1.byteCapacity = 104857600
app.sinks.k1.type = avro
app.sinks.k1.hostname = 172.19.100.92
app.sinks.k1.port = 44466
app.sinks.k2.type = avro
app.sinks.k2.hostname = 172.19.100.102
app.sinks.k2.port = 44466
app.sources.r1.channels = c1
app.sinks.k1.channel = c1
app.sinks.k2.channel = c1
app.sinkgroups = g1
app.sinkgroups.g1.sinks = k1 k2
app.sinkgroups.g1.processor.type = load_balance
app.sinkgroups.g1.processor.backoff = true
app.sinkgroups.g1.processor.selector = round_robin
-------------------------------------------------------------------------------------
批量替换命令参考:
#修改flume-env.sh
sed -i 's/172.18.100.159/172.19.100.73/g;s/S000/S001/g' /home/jyapp/flume-agent/conf/flume-env.sh
#修改app.conf
sed -i 's/172.18.100.159/172.19.100.73/g;s/S000/S001/g' /home/jyapp/flume-agent/conf/app.conf
后台启动agent
cd /home/jyapp/flume-agent
bin/flume-ng agent --conf conf --conf-file conf/app.conf --name app &
停止agent,使用kill命令
- apache flume agent安装
- apache flume collector 安装
- apache flume-ng做agent拦截器可以输出IP
- apache-flume-1.5.0.1 安装
- apache flume单机安装、运行
- apache-flume-1.7.x配置安装
- Apache Flume 1.7.0 配置安装
- Apache-flume-1.6.0安装配置
- apache-flume-1.7.0安装及测试
- apache-flume-1.7.x配置安装
- Apache Flume
- Apache Flume
- Apache Flume
- Apache Flume
- Flume-ng agent配置说明
- flume agent arvo传数据
- Flume日志采集多级Agent
- flume的多agent连接
- Android屏幕适配相关知识
- 设计模式之二---观察者设计模式
- 从内存DC中保存成叠加的bmp文件
- hdu1877 又一版 A+B (栈)
- A Beginner's Guide To LVM [2/9]
- apache flume agent安装
- 【BZOJ3875】[Ahoi2014]骑士游戏【最短路】【DP】
- [Android 分享] View 和 ViewGroup 事件传递机制
- CSDN写博客怎么添加代码片,让博客更美观
- iOS开发------内存管理
- Poisson regression fitted by glm(), maximum likelihood, and MCMC
- express 4.x.x 文档翻译
- 牛课网 | 选择
- C/C++中strlen的用法