Flume 收集Nginx日志到Hdfs Tail-to-hdfs sink
来源:互联网 发布:什么是网络故障诊断 编辑:程序博客网 时间:2024/05/22 14:03
nginx,access.log日志约8000条/s,每100w条数据约253M,需要2min
agent1.sources = source1agent1.sinks = sink1agent1.channels = channel1# Describe/configure spooldir source1#agent1.sources.source1.type = spooldir#agent1.sources.source1.spoolDir = /var/log/apache/flumeSpool1#agent1.sources.source1.fileHeader = true# Describe/configure tail -F source1agent1.sources.source1.type = exec agent1.sources.source1.command = tail -F /tmp/log.logagent1.sources.source1.channels = channel1# Describe/configure nc source1#agent1.sources.source1.type = netcat#agent1.sources.source1.bind = localhost#agent1.sources.source1.port = 44444#configure host for sourceagent1.sources.source1.interceptors = i1agent1.sources.source1.interceptors.i1.type = hostagent1.sources.source1.interceptors.i1.hostHeader = hostname# Describe sink1#agent1.sinks.sink1.type = loggeragent1.sinks.sink1.type = hdfs#a1.sinks.k1.channel = c1#agent1.sinks.sink1.hdfs.path =hdfs://xxx:9000/tmp/tail/%y-%m-%d/%H%M%Sagent1.sinks.sink1.hdfs.path =hdfs://xxx:9000/tmp/tail/%y-%m-%d/%Hagent1.sinks.sink1.hdfs.filePrefix = %{hostname}/events-agent1.sinks.sink1.hdfs.maxOpenFiles = 5000 agent1.sinks.sink1.hdfs.batchSize= 500agent1.sinks.sink1.hdfs.fileType = DataStreamagent1.sinks.sink1.hdfs.writeFormat =Textagent1.sinks.sink1.hdfs.rollSize = 0agent1.sinks.sink1.hdfs.rollCount = 1000000agent1.sinks.sink1.hdfs.rollInterval = 600#agent1.sinks.sink1.hdfs.round = true#agent1.sinks.sink1.hdfs.roundValue = 10#agent1.sinks.sink1.hdfs.roundUnit = minuteagent1.sinks.sink1.hdfs.useLocalTimeStamp = true# Use a channel which buffers events in memoryagent1.channels.channel1.type = memoryagent1.channels.channel1.keep-alive = 120agent1.channels.channel1.capacity = 500000agent1.channels.channel1.transactionCapacity = 600# Bind the source and sink to the channelagent1.sources.source1.channels = channel1agent1.sinks.sink1.channel = channel1
0 0
- Flume 收集Nginx日志到Hdfs Tail-to-hdfs sink
- Flume 收集Nginx日志到Hdfs Tail-to-hdfs sink
- flume 收集日志到HDFS
- Flume 日志收集、使用Flume收集日志到HDFS
- Flume 收集Nginx日志 tail-to-avro avro-to-file
- Flume 收集Nginx日志 tail-to-avro avro-to-file
- flume sink hdfs异常
- Flume之——采集Nginx日志到HDFS
- Flume中的HDFS Sink配置
- Flume<HDFS Sink参数配置>
- Flume HDFS sink 配置手册
- Flume中的HDFS Sink配置
- Flume-NG + HDFS + HIVE 日志收集分析
- Flume-NG + HDFS + HIVE 日志收集分析
- Flume NG flume-hdfs-sink 源代码分析
- flume的导日志数据到hdfs
- 理解flume hdfs sink round 和roll
- Flume HDFS Sink使用及源码分析
- windows环境使用python google protobuf
- FastStone Capture 7.7 截图工具下载地址及使用
- 广州传智播客网页平面学员感想
- IOS7.1下使用AdHoc方法下载的解决方案
- ADO.net入门(二)快快乐乐走婆家(连接到数据库)
- Flume 收集Nginx日志到Hdfs Tail-to-hdfs sink
- 什么是GNU?
- MVC Razor 语法
- “玩转”Java系列—CSS层叠样式表
- Could not load file or assembly 'System.Web.Helpers...错误
- linux 单引号,双引号,反引号
- 全景图片(鱼眼)的平面映射矫正
- android abd shell 命令小结
- VS2010/VS2012编译Stlport 5.2.1