Flume把网络流量存入日志文件
来源:互联网 发布:手机听歌软件 编辑:程序博客网 时间:2024/05/21 22:32
1.创建agent配置文件
把下列内容存入agent1.conf,并保存到Flume的工作目录/opt/flume/bin下面
agent1.sources = netsource
agent1.sinks = logsink
agent1.channels = memorychannel
agent1.sources.netsource.type = netcat
agent1.sources.netsource.bind = localhost
agent1.sources.netsource.port = 3000
agent1.sinks.logsink.type = logger
agent1.channels.memorychannel.type = memory
agent1.channels.memorychannel.capacity = 1000
agent1.channels.memorychannel.transactionCapacity = 100
agent1.sources.netsource.channels = memorychannel
agent1.sinks.logsink.channel = memorychannel
2.启动一个flume代理
caiyong@caiyong:/opt/flume/bin$ flume-ng agent --conf conf --conf-file agent1.conf --name agent1
Info: Including Hadoop libraries found via (/opt/hadoop/bin/hadoop) for HDFS access
Info: Excluding /opt/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from classpath
Info: Excluding /opt/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from classpath
+ exec /usr/java/jdk/bin/java -Xmx20m -cp 'conf:/opt/flume/lib/*:/opt/hadoop/conf:/usr/java/jdk/lib/tools.jar:/opt/hadoop/libexec/..:/opt/hadoop/libexec/../hadoop-core-1.2.1.jar:/opt/hadoop/libexec/../lib/asm-3.2.jar:/opt/hadoop/libexec/../lib/aspectjrt-1.6.11.jar:/opt/hadoop/libexec/../lib/aspectjtools-1.6.11.jar:/opt/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/opt/hadoop/libexec/../lib/commons-cli-1.2.jar:/opt/hadoop/libexec/../lib/commons-codec-1.4.jar:/opt/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/opt/hadoop/libexec/../lib/commons-configuration-1.6.jar:/opt/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/libexec/../lib/commons-digester-1.8.jar:/opt/hadoop/libexec/../lib/commons-el-1.0.jar:/opt/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/opt/hadoop/libexec/../lib/commons-io-2.1.jar:/opt/hadoop/libexec/../lib/commons-lang-2.4.jar:/opt/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/opt/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/libexec/../lib/commons-math-2.1.jar:/opt/hadoop/libexec/../lib/commons-net-3.1.jar:/opt/hadoop/libexec/../lib/core-3.1.1.jar:/opt/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.2.1.jar:/opt/hadoop/libexec/../lib/hadoop-fairscheduler-1.2.1.jar:/opt/hadoop/libexec/../lib/hadoop-thriftfs-1.2.1.jar:/opt/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/opt/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/opt/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/opt/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/libexec/../lib/jdeb-0.8.jar:/opt/hadoop/libexec/../lib/jersey-core-1.8.jar:/opt/hadoop/libexec/../lib/jersey-json-1.8.jar:/opt/hadoop/libexec/../lib/jersey-server-1.8.jar:/opt/hadoop/libexec/../lib/jets3t-0.6.1.jar:/opt/hadoop/libexec/../lib/jetty-6.1.26.jar:/opt/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/opt/hadoop/libexec/../lib/jsch-0.1.42.jar:/opt/hadoop/libexec/../lib/junit-4.5.jar:/opt/hadoop/libexec/../lib/kfs-0.2.2.jar:/opt/hadoop/libexec/../lib/log4j-1.2.15.jar:/opt/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/opt/hadoop/libexec/../lib/oro-2.0.8.jar:/opt/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/libexec/../lib/xmlenc-0.52.jar:/opt/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar' -Djava.library.path=:/opt/hadoop/libexec/../lib/native/Linux-i386-32 org.apache.flume.node.Application --conf-file agent1.conf --name agent1
15/03/14 13:11:00 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
15/03/14 13:11:00 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:agent1.conf
15/03/14 13:11:00 INFO conf.FlumeConfiguration: Processing:logsink
15/03/14 13:11:00 INFO conf.FlumeConfiguration: Processing:logsink
15/03/14 13:11:00 INFO conf.FlumeConfiguration: Added sinks: logsink Agent: agent1
15/03/14 13:11:00 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1]
15/03/14 13:11:00 INFO node.AbstractConfigurationProvider: Creating channels
15/03/14 13:11:00 INFO channel.DefaultChannelFactory: Creating instance of channel memorychannel type memory
15/03/14 13:11:00 INFO node.AbstractConfigurationProvider: Created channel memorychannel
15/03/14 13:11:00 INFO source.DefaultSourceFactory: Creating instance of source netsource, type netcat
15/03/14 13:11:00 INFO sink.DefaultSinkFactory: Creating instance of sink: logsink, type: logger
15/03/14 13:11:00 INFO node.AbstractConfigurationProvider: Channel memorychannel connected to [netsource, logsink]
15/03/14 13:11:01 INFO node.Application: Starting new configuration:{ sourceRunners:{netsource=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:netsource,state:IDLE} }} sinkRunners:{logsink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@1b688c0 counterGroup:{ name:null counters:{} } }} channels:{memorychannel=org.apache.flume.channel.MemoryChannel{name: memorychannel}} }
15/03/14 13:11:01 INFO node.Application: Starting Channel memorychannel
15/03/14 13:11:01 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: memorychannel: Successfully registered new MBean.
15/03/14 13:11:01 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memorychannel started
15/03/14 13:11:01 INFO node.Application: Starting Sink logsink
15/03/14 13:11:01 INFO node.Application: Starting Source netsource
15/03/14 13:11:01 INFO source.NetcatSource: Source starting
15/03/14 13:11:01 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:3000]
3.在另一个窗口中,远程连接至本地主机的3000端口,然后输入一些文本
caiyong@caiyong:~$ telnet localhost 3000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
hello
OK
this is a test
OK
再在原来的窗口可以看到末尾增加了如下数据:
15/03/14 13:11:41 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 20 0D hello . }
15/03/14 13:11:47 INFO sink.LoggerSink: Event: { headers:{} body: 74 68 69 73 20 69 73 20 61 20 74 65 73 74 0D this is a test. }
把下列内容存入agent1.conf,并保存到Flume的工作目录/opt/flume/bin下面
agent1.sources = netsource
agent1.sinks = logsink
agent1.channels = memorychannel
agent1.sources.netsource.type = netcat
agent1.sources.netsource.bind = localhost
agent1.sources.netsource.port = 3000
agent1.sinks.logsink.type = logger
agent1.channels.memorychannel.type = memory
agent1.channels.memorychannel.capacity = 1000
agent1.channels.memorychannel.transactionCapacity = 100
agent1.sources.netsource.channels = memorychannel
agent1.sinks.logsink.channel = memorychannel
2.启动一个flume代理
caiyong@caiyong:/opt/flume/bin$ flume-ng agent --conf conf --conf-file agent1.conf --name agent1
Info: Including Hadoop libraries found via (/opt/hadoop/bin/hadoop) for HDFS access
Info: Excluding /opt/hadoop/libexec/../lib/slf4j-api-1.4.3.jar from classpath
Info: Excluding /opt/hadoop/libexec/../lib/slf4j-log4j12-1.4.3.jar from classpath
+ exec /usr/java/jdk/bin/java -Xmx20m -cp 'conf:/opt/flume/lib/*:/opt/hadoop/conf:/usr/java/jdk/lib/tools.jar:/opt/hadoop/libexec/..:/opt/hadoop/libexec/../hadoop-core-1.2.1.jar:/opt/hadoop/libexec/../lib/asm-3.2.jar:/opt/hadoop/libexec/../lib/aspectjrt-1.6.11.jar:/opt/hadoop/libexec/../lib/aspectjtools-1.6.11.jar:/opt/hadoop/libexec/../lib/commons-beanutils-1.7.0.jar:/opt/hadoop/libexec/../lib/commons-beanutils-core-1.8.0.jar:/opt/hadoop/libexec/../lib/commons-cli-1.2.jar:/opt/hadoop/libexec/../lib/commons-codec-1.4.jar:/opt/hadoop/libexec/../lib/commons-collections-3.2.1.jar:/opt/hadoop/libexec/../lib/commons-configuration-1.6.jar:/opt/hadoop/libexec/../lib/commons-daemon-1.0.1.jar:/opt/hadoop/libexec/../lib/commons-digester-1.8.jar:/opt/hadoop/libexec/../lib/commons-el-1.0.jar:/opt/hadoop/libexec/../lib/commons-httpclient-3.0.1.jar:/opt/hadoop/libexec/../lib/commons-io-2.1.jar:/opt/hadoop/libexec/../lib/commons-lang-2.4.jar:/opt/hadoop/libexec/../lib/commons-logging-1.1.1.jar:/opt/hadoop/libexec/../lib/commons-logging-api-1.0.4.jar:/opt/hadoop/libexec/../lib/commons-math-2.1.jar:/opt/hadoop/libexec/../lib/commons-net-3.1.jar:/opt/hadoop/libexec/../lib/core-3.1.1.jar:/opt/hadoop/libexec/../lib/hadoop-capacity-scheduler-1.2.1.jar:/opt/hadoop/libexec/../lib/hadoop-fairscheduler-1.2.1.jar:/opt/hadoop/libexec/../lib/hadoop-thriftfs-1.2.1.jar:/opt/hadoop/libexec/../lib/hsqldb-1.8.0.10.jar:/opt/hadoop/libexec/../lib/jackson-core-asl-1.8.8.jar:/opt/hadoop/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/opt/hadoop/libexec/../lib/jasper-compiler-5.5.12.jar:/opt/hadoop/libexec/../lib/jasper-runtime-5.5.12.jar:/opt/hadoop/libexec/../lib/jdeb-0.8.jar:/opt/hadoop/libexec/../lib/jersey-core-1.8.jar:/opt/hadoop/libexec/../lib/jersey-json-1.8.jar:/opt/hadoop/libexec/../lib/jersey-server-1.8.jar:/opt/hadoop/libexec/../lib/jets3t-0.6.1.jar:/opt/hadoop/libexec/../lib/jetty-6.1.26.jar:/opt/hadoop/libexec/../lib/jetty-util-6.1.26.jar:/opt/hadoop/libexec/../lib/jsch-0.1.42.jar:/opt/hadoop/libexec/../lib/junit-4.5.jar:/opt/hadoop/libexec/../lib/kfs-0.2.2.jar:/opt/hadoop/libexec/../lib/log4j-1.2.15.jar:/opt/hadoop/libexec/../lib/mockito-all-1.8.5.jar:/opt/hadoop/libexec/../lib/oro-2.0.8.jar:/opt/hadoop/libexec/../lib/servlet-api-2.5-20081211.jar:/opt/hadoop/libexec/../lib/xmlenc-0.52.jar:/opt/hadoop/libexec/../lib/jsp-2.1/jsp-2.1.jar:/opt/hadoop/libexec/../lib/jsp-2.1/jsp-api-2.1.jar' -Djava.library.path=:/opt/hadoop/libexec/../lib/native/Linux-i386-32 org.apache.flume.node.Application --conf-file agent1.conf --name agent1
15/03/14 13:11:00 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
15/03/14 13:11:00 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:agent1.conf
15/03/14 13:11:00 INFO conf.FlumeConfiguration: Processing:logsink
15/03/14 13:11:00 INFO conf.FlumeConfiguration: Processing:logsink
15/03/14 13:11:00 INFO conf.FlumeConfiguration: Added sinks: logsink Agent: agent1
15/03/14 13:11:00 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1]
15/03/14 13:11:00 INFO node.AbstractConfigurationProvider: Creating channels
15/03/14 13:11:00 INFO channel.DefaultChannelFactory: Creating instance of channel memorychannel type memory
15/03/14 13:11:00 INFO node.AbstractConfigurationProvider: Created channel memorychannel
15/03/14 13:11:00 INFO source.DefaultSourceFactory: Creating instance of source netsource, type netcat
15/03/14 13:11:00 INFO sink.DefaultSinkFactory: Creating instance of sink: logsink, type: logger
15/03/14 13:11:00 INFO node.AbstractConfigurationProvider: Channel memorychannel connected to [netsource, logsink]
15/03/14 13:11:01 INFO node.Application: Starting new configuration:{ sourceRunners:{netsource=EventDrivenSourceRunner: { source:org.apache.flume.source.NetcatSource{name:netsource,state:IDLE} }} sinkRunners:{logsink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@1b688c0 counterGroup:{ name:null counters:{} } }} channels:{memorychannel=org.apache.flume.channel.MemoryChannel{name: memorychannel}} }
15/03/14 13:11:01 INFO node.Application: Starting Channel memorychannel
15/03/14 13:11:01 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: memorychannel: Successfully registered new MBean.
15/03/14 13:11:01 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: memorychannel started
15/03/14 13:11:01 INFO node.Application: Starting Sink logsink
15/03/14 13:11:01 INFO node.Application: Starting Source netsource
15/03/14 13:11:01 INFO source.NetcatSource: Source starting
15/03/14 13:11:01 INFO source.NetcatSource: Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:3000]
3.在另一个窗口中,远程连接至本地主机的3000端口,然后输入一些文本
caiyong@caiyong:~$ telnet localhost 3000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
hello
OK
this is a test
OK
再在原来的窗口可以看到末尾增加了如下数据:
15/03/14 13:11:41 INFO sink.LoggerSink: Event: { headers:{} body: 68 65 6C 6C 6F 20 0D hello . }
15/03/14 13:11:47 INFO sink.LoggerSink: Event: { headers:{} body: 74 68 69 73 20 69 73 20 61 20 74 65 73 74 0D this is a test. }
0 0
- Flume把网络流量存入日志文件
- Flume 抓取日志文件存入MySQL中
- flume-ng抓取日志文件存入mysql中
- 把文件存入数据库
- 把数据存入文件
- Flume实战采集文件内容存入HDFS
- 把图片文件存入数据库
- 把图片文件存入数据库
- PHP: 把数组存入文件
- 模拟使用Flume监听日志变化,并且把增量的日志文件写入到hdfs中
- iOS将日志存入文件
- flume+kafka读取日志文件
- Flume采集处理日志文件
- VB把文件存入数据库IMAGE字段
- 生成项目异常日志存入xml文件
- python 将增量文件日志存入数据库
- Flume向hdfs发送日志文件配置
- Flume-NG 日志文件收集 简单实例
- MongoDB高级查询用法大全 - 程序猿_哲
- 2015.03.11 vim操作+特殊权限
- 常量折叠问题
- maven 服务搭建和使用
- 在RHEL6.0 X64系统中安装oracle 11g数据库及安装后设置
- Flume把网络流量存入日志文件
- Hbase 常用工具类
- BB平台oracle11g-180天密码过期问题处理
- 《一直以来伴随我的一些学习习惯》的记录
- Hadoop、Spark、HBase与Redis的适用性讨论
- android handler消息机制
- Red Hat搭建邮件服务器(sendmail+dovecot+openwebmail+procmail)
- maven 工程简单打包
- 函数调用的执行过程