使用Log4j将日志实时写入Flume

需求：使用Flume实时收集WebServer或者其它服务器上通过log4j产生的日志

Agent选型：Log4jAppender的使用

创建一个maven项目，pom.xml文件中引入依赖

<dependency>    <groupId>log4j</groupId>    <artifactId>log4j</artifactId>    <version>1.2.16</version></dependency><dependency>    <groupId>org.slf4j</groupId>    <artifactId>slf4j-log4j12</artifactId>    <version>1.7.21</version>    <scope>test</scope></dependency>

在src/main/resource目录下新建log4j.properties，配置如下

log4j.rootLogger=INFO,stdoutlog4j.appender.stdout = org.apache.log4j.ConsoleAppenderlog4j.appender.stdout.target = System.outlog4j.appender.stdout.layout=org.apache.log4j.PatternLayoutlog4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} [%t] [%c] [%p] - %m%n

将resource目录加载到classpath下

选择左侧Modules，找到resource目录，点击上面的Resources

新建LoggerGenerate类，用于产生日志（模仿项目中的日志输出）

import org.apache.log4j.Logger;/** * 循环打印值，模仿日志输出 */public class LoggerGenerate {    private static Logger logger = Logger.getLogger(LoggerGenerate.class.getName());    public static void main(String[] args) throws InterruptedException {        int index = 0;        while (true) {            Thread.sleep(1000);            logger.info("value is：" + index++);        }    }}

运行测试（部分结果）：

2017-9-25 09:44:45,238 [main] [LoggerGenerate] [INFO] - value is：02017-9-25 09:44:46,241 [main] [LoggerGenerate] [INFO] - value is：12017-9-25 09:44:47,241 [main] [LoggerGenerate] [INFO] - value is：22017-9-25 09:44:48,241 [main] [LoggerGenerate] [INFO] - value is：32017-9-25 09:44:49,255 [main] [LoggerGenerate] [INFO] - value is：42017-9-25 09:44:50,258 [main] [LoggerGenerate] [INFO] - value is：5

目前这些日志信息都打印在了控制台，接下来我们想要把这些日志收集到flume，首先先配置Agent

# Name the components on this agentflume-log-agent.sources = arvo-sourceflume-log-agent.sinks = logger-sinkflume-log-agent.channels = memory-channel# Describe/configure the sourceflume-log-agent.sources.arvo-source.type = avroflume-log-agent.sources.arvo-source.bind = Masterflume-log-agent.sources.arvo-source.port = 44444# Describe the sinkflume-log-agent.sinks.logger-sink.type = logger# Use a channel which buffers events in memoryflume-log-agent.channels.memory-channel.type = memory# Bind the source and sink to the channelflume-log-agent.sources.arvo-source.channels = memory-channelflume-log-agent.sinks.logger-sink.channel = memory-channel

启动Agent

flume-ng agent \--name flume-log-agent \--conf $FLUME_HOME/conf \--conf-file $FLUME_HOME/config/flume-log-agent.conf \-Dflume.root.logger=INFO,console

想要将控制台的日志收集到flume，需要修改log4j.properties中的配置
参考官网：http://flume.apache.org/FlumeUserGuide.html

log4j.properties中新增内容如下配置

log4j.appender.flume = org.apache.flume.clients.log4jappender.Log4jAppenderlog4j.appender.flume.Hostname = Masterlog4j.appender.flume.Port = 44444log4j.appender.flume.UnsafeMode = true

在 log4j.rootLogger=INFO,stdout 后面加上flume

log4j.rootLogger=INFO,stdout,flume

运行测试，笔者的项目是在windows下创建的（你也可以将项目部署在linux服务器中），需要在windows的hosts文件中配置映射，否者在log4j中是不可以写主机名的，只能写ip地址，参照下图配置即可

在hosts文件中添加一行内容（格式：IP 主机名）

192.168.242.150 Master

在运行期间，报了一个错误，如下
java.lang.ClassNotFoundException: org.apache.flume.clients.log4jappender.Log4jAppender

报错的原因是类找不到，很明显是项目中没有添加依赖，在pom.xml文件中添加flume-ng-log4jappender的依赖

<dependency>    <groupId>org.apache.flume.flume-ng-clients</groupId>    <artifactId>flume-ng-log4jappender</artifactId>    <version>1.6.0</version></dependency>

再次运行项目，查看Agent控制台中打印的信息

2017-09-25 10:16:12,862 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926512143, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 30             value is...0 }2017-09-25 10:16:12,863 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926513197, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 31             value is...1 }2017-09-25 10:16:12,864 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926514203, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 32             value is...2 }2017-09-25 10:16:13,082 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926515210, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 33             value is...3 }2017-09-25 10:16:14,092 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926516221, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 34             value is...4 }2017-09-25 10:16:15,103 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{flume.client.log4j.timestamp=1508926517228, flume.client.log4j.logger.name=LoggerGenerate, flume.client.log4j.log.level=20000, flume.client.log4j.message.encoding=UTF8} body: 76 61 6C 75 65 20 69 73 EF BC 9A 35             value is...5 }

总结：在Agent的配置文件编写过程中，常用的参数设置，无需死记硬背（参照官网即可）