Flume 安装, 测试
来源:互联网 发布:邢山虎 知乎 编辑:程序博客网 时间:2024/05/17 08:05
1. 安装
选择适合自己的版本:[下载](https://flume.apache.org/download.html)
2. 解压
gunzip flume-ng-1.6.0-cdh5.5.1.tar.gz
3. 测试
3.1 mkdir test 3.2 cd test && mkdir logs 3.3 touch flume-conf.properties
test.sources = s1
test.channels = c1
test.sinks = s1
test.sources.s1.type = TAILDIR
test.sources.s1.filegroups = t1
test.sources.s1.filegroups.t1= /data/flume/test/test.*.log
test.sources.s1.batchSize = 100
test.sources.s1.positionFile = /data/flume/test/position.json
test.sources.s1.channels = c1
test.sources.s1.writePosInterval = 5000
test.sources.s1.skipToEnd = false
test.channels.c1.type = SpillableMemory
test.channels.c1.memoryCapacity = 100000
test.channels.c1.byteCapacity = 134217728
test.channels.c1.checkpointDir = /data/flume/test/checkpoint
test.channels.c1.dataDirs = /data/flume/test/data
test.sinks.s1.type = logger
test.sinks.s1.channel = c1
3.4 touch flume-env.sh
HTTP_MONITORING_PORT=34520
export JAVA_OPTS=”$JAVA_OPTS -Xms256m -Xmx256m -Dflume.project.dir=$opt_conf -Dflume.monitoring.type=http -Dflume.monitoring.port=\$HTTP_MONITORING_PORT”
3.5 touch log4j.properties
flume.root.logger=INFO,LOGFILE
flume.log.dir=${flume.project.dir}/logs
flume.log.file=flume.log
log4j.logger.org.apache.flume.lifecycle = INFO
log4j.logger.org.jboss = WARN
log4j.logger.org.mortbay = INFO
log4j.logger.org.apache.avro.ipc.NettyTransceiver = WARN
log4j.logger.org.apache.hadoop = INFO
log4j.logger.org.apache.hadoop.hive = ERROR
log4j.rootLogger=${flume.root.logger}
log4j.appender.LOGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.LOGFILE.MaxFileSize=100MB
log4j.appender.LOGFILE.MaxBackupIndex=10
log4j.appender.LOGFILE.File=${flume.log.dir}/${flume.log.file}
log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.LOGFILE.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n
log4j.appender.DAILY=org.apache.log4j.rolling.RollingFileAppender
log4j.appender.DAILY.rollingPolicy=org.apache.log4j.rolling.TimeBasedRollingPolicy
log4j.appender.DAILY.rollingPolicy.ActiveFileName=${flume.log.dir}/${flume.log.file}
log4j.appender.DAILY.rollingPolicy.FileNamePattern=${flume.log.dir}/${flume.log.file}.%d{yyyy-MM-dd}
log4j.appender.DAILY.layout=org.apache.log4j.PatternLayout
log4j.appender.DAILY.layout.ConversionPattern=%d{dd MMM yyyy HH:mm:ss,SSS} %-5p [%t] (%C.%M:%L) %x - %m%n
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d (%t) [%p - %l] %m%n
4. 启动
/data/flume/apache-flume-1.6.0-cdh5.5.1-bin/bin/flume-ng agent -name test --conf . --conf-file flume-conf.propertiestail -f logs/flume.log
- Flume安装与测试
- Flume安装测试
- Flume 安装, 测试
- Flume安装及测试
- Flume 安装,简单测试
- Flume单机安装并且测试
- Flume的安装和测试
- Flume的安装和测试
- flume-ng+kafka单机安装测试
- flume安装及source、sink、Channels测试
- linux安装flume和集成kafka测试
- apache-flume-1.7.0安装及测试
- Flume-安装Flume
- flume 安装
- flume安装
- flume 安装
- flume 安装
- Flume安装
- linux 下CodeBlocks 安装
- 获取数字证书信息
- Android 画虚线的问题
- JAVA开发11--获取系统当前时间和简单的时间差
- 5.导航条、分页导航
- Flume 安装, 测试
- mac修改mysql密码
- FindBugs 插件使用
- ActiveMQ的启动和配置介绍
- 数据结构第四次作业(二叉树的基本操作实现)
- 树和二叉树
- 引用和指针的区别
- iOS视频播放的四种方案
- 智力游戏