flume配置启动

功能

自动监控日志变化

核心组件

Source : 完成对日志数据的收集，分成transtion 和 event 打入到channel之中。
Channel : 主要提供一个队列的功能，对source提供中的数据进行简单的缓存。
Sink : 取出Channel中的数据，进行相应的存储文件系统，数据库，或者提交到远程服务器

学习核心：配置个性化-组件的类型

Flume（2）组件概述与列表

Flume 入门–几种不同的Sources

错误:找不到或无法加载主类 org.apache.flume.tools.GetJavaProperty

原因：hbase的配置

解决办法

配置文件示例1-spooldir

# Licensed to the Apache Software Foundation (ASF) under one# or more contributor license agreements.  See the NOTICE file# distributed with this work for additional information# regarding copyright ownership.  The ASF licenses this file# to you under the Apache License, Version 2.0 (the# "License"); you may not use this file except in compliance# with the License.  You may obtain a copy of the License at##  http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing,# software distributed under the License is distributed on an# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY# KIND, either express or implied.  See the License for the# specific language governing permissions and limitations# under the License.# The configuration file needs to define the sources, # the channels and the sinks.# Sources, channels and sinks are defined per agent, # in this case called 'agent'agent1.sources = avro-source1agent1.channels = ch1agent1.sinks = logger-sink1# sourcesagent1.sources.avro-source1.type = spooldiragent1.sources.avro-source1.channels = ch1agent1.sources.avro-source1.spoolDir = /home/shuai/logs/agent1.sources.avro-source1.fileHeader = trueagent1.sources.avro-source1.bind = 0.0.0.0agent1.sources.avro-source1.port = 4141# sinkagent1.sinks.logger-sink1.type = loggeragent1.sinks.logger-sink1.channel = ch1# channelagent1.channels.ch1.type = memoryagent1.channels.ch1.capacity = 1000agent1.channels.ch1.transactionCapacity = 100

启动

参考教程启动[Flume]安装，部署与应用案例

./flume-ng agent –conf ../conf –conf-file ../conf/flume-spool.conf –name agent -Dflume.root.logger=INFO,console

没有提示报错，但是没什么功能

参数 作用 列表 –conf 或 -c 指定配置文件夹，包含flume-env.sh和log4j的配置文件 –conf ../conf –conf-file 或 -f 配置文件地址 –conf-file ../conf/flume.conf –name 或 -n agent名称 –name a1 -z zookeeper连接字符串 -z zkhost:2181,zkhost1:2181 -p zookeeper中的存储路径前缀 -p /flume

分析日志

SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/data/hadoop/apache-flume-1.8.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/data/hadoop/hadoop-2.7.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/data/hadoop/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.2017-10-15 21:44:02,380 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start(PollingPropertiesFileConfigurationProvider.java:62)] Configuration provider starting2017-10-15 21:44:02,382 (conf-file-poller-0) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:134)] Reloading configuration file:../conf/flume-spool.conf2017-10-15 21:44:02,385 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:logger-sink12017-10-15 21:44:02,386 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:930)] Added sinks: logger-sink1 Agent: agent12017-10-15 21:44:02,386 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:1016)] Processing:logger-sink12017-10-15 21:44:02,392 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:140)] Post-validation flume configuration contains configuration for agents: [agent1]2017-10-15 21:44:02,392 (conf-file-poller-0) [WARN - org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:135)] No configuration found for this host:agent2017-10-15 21:44:02,396 (conf-file-poller-0) [INFO - org.apache.flume.node.Application.startAllComponents(Application.java:137)] Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }

agent的名字是agent1,改成agent1成功

./flume-ng agent --conf ../conf --conf-file ../conf/flume-spool.conf --name agent1 -Dflume.root.logger=INFO,console

配置文件示例2-spooldir

# Licensed to the Apache Software Foundation (ASF) under one# or more contributor license agreements.  See the NOTICE file# distributed with this work for additional information# regarding copyright ownership.  The ASF licenses this file# to you under the Apache License, Version 2.0 (the# "License"); you may not use this file except in compliance# with the License.  You may obtain a copy of the License at##  http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing,# software distributed under the License is distributed on an# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY# KIND, either express or implied.  See the License for the# specific language governing permissions and limitations# under the License.# The configuration file needs to define the sources, # the channels and the sinks.# Sources, channels and sinks are defined per agent, # in this case called 'agent'agent.sources = seqGenSrcagent.channels = memoryChannelagent.sinks = loggerSink# For each one of the sources, the type is definedagent.sources.seqGenSrc.type = spooldiragent.sources.seqGenSrc.spoolDir = /data/nginx/logs# The channel can be defined as follows.agent.sources.seqGenSrc.channels = memoryChannel# Each sink's type must be definedagent.sinks.loggerSink.type = logger#Specify the channel the sink should useagent.sinks.loggerSink.channel = memoryChannel# Each channel's type is defined.agent.channels.memoryChannel.type = memory# Other config values specific to each type of channel(sink or source)# can be defined as well# In this case, it specifies the capacity of the memory channelagent.channels.memoryChannel.capacity = 1000agent.channels.memoryChannel.transactionCapacity = 100

采用flume提供的默认模板，需要将source 的类型改成spooldir并添加监控的文件夹，此时的名称必须为agent才行
➜ bin ./flume-ng agent –conf ../conf –conf-file ../conf/flume-conf.properties –name agent -Dflume.root.logger=INFO,console