Flume之Log4JAppender接收Log4j日志

简介

Log4JAppender和LoadBalancingLog4jAppender可以将应用服务器的日志通过AvroSource实时的把日志传输到日志服务器，然后在传输到监控系统或者是HDFS中存储，FLume的Log4JAppender和LoadBalancingLog4jAppender必须使用Log4j的异步加载器，否则日志服务器down机，将会导致应用服务器异常，影响线上环境的使用。 应用程序如果要用Log4JAppender就要依赖相关的jar，在应用服务器可以通过如下Maven配置获取到Log4JAppender的jar和log4j的相关jar：

<!-- flume log4j appender start --><dependency>    <groupId>org.apache.flume.flume-ng-clients</groupId>    <artifactId>flume-ng-log4jappender</artifactId>    <version>1.6.0</version></dependency><!-- flume log4j appender end --><!-- log4j start --><dependency>      <groupId>log4j</groupId>      <artifactId>log4j</artifactId>      <version>1.2.17</version>  </dependency>  <dependency>      <groupId>org.slf4j</groupId>      <artifactId>slf4j-api</artifactId>      <version>1.7.5</version>  </dependency>  <dependency>      <groupId>org.slf4j</groupId>      <artifactId>slf4j-log4j12</artifactId>      <version>1.7.5</version>  </dependency><!-- log4j end -->

Log4jAppender和LoadBalancingLog4jAppender有如下区别：

1，LoadBalancingLog4jAppender支持一组AvroSource。
2，LoadBalancingLog4jAppender支持round-robin，random或自定义的负载均衡方式。
3，当Event发送失败的情况下，LoadBalancingLog4jAppender支持backoff策略。

Log4JAppender配置文件

属性名默认描述Hostname–运行在远程Flume Agent中Avro Source的HostnamePort–运行在远程Flume Agent中Avro Source的监听端口UnsafeModefalse如果设置为true，那么该Appender在发送Event失败不会抛出异常。AvroReflectionEnabledfalse使用Avro反射机制序列化Log4j的Event事件。（如果日志是字符串就不需要了。）AvroSchemaUrl–一个可以检索Avro schema的URL

LoadBalancingLog4jAppender配置文件

属性名默认描述Hosts–通过AvroSource监听Event的host:port列表，用空格分隔SelectorROUND_ROBIN      
选择机制。选择机制包含：ROUND_ROBIN，RANDOM或者是自定义MaxBackoff–MaxBackoff是一个long型值，代表一个节点消费一个event失败的情况下，负载均衡客户端最大的backoff时间（单位:毫秒），默认不启动backoffUnsafeModefalse如果设置为true，那么该Appender在发送Event失败不会抛出异常。AvroReflectionEnabledfalse使用Avro反射机制序列化Log4j的Event事件。（如果日志是字符串就不需要了。）AvroSchemaUrl–一个可以检索Avro schema的URL

示例

应用程序端log4j配置

一，采用Log4JAppender

1，采用properties方式：

log4j.appender.flume=org.apache.flume.clients.log4jappender.Log4jAppenderlog4j.appender.flume.Hostname=10.0.1.75log4j.appender.flume.Port=5555log4j.appender.flume.UnsafeMode=truelog4j.appender.flume.layout=org.apache.log4j.PatternLayoutlog4j.appender.flume.layout.ConversionPattern=%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n

2，采用xml方式：

<appender name="flume" class="org.apache.flume.clients.log4jappender.Log4jAppender"><param name="Hostname" value="10.0.1.75" /><param name="Port" value="5555" />        <param name="UnsafeMode" value="true" /><layout class="org.apache.log4j.PatternLayout"><param name="ConversionPattern" value="%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n" /></layout></appender>

二，采用LoadBalancingLog4jAppender

1，采用properties方式：

log4j.appender.flume=org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppenderlog4j.appender.flume.Hosts=10.0.1.75:5555 10.0.1.76:5555 10.0.1.77:5555log4j.appender.flume.UnsafeMode=truelog4j.appender.flume.MaxBackoff=30000log4j.appender.flume.Selector=RANDOMlog4j.appender.flume.layout=org.apache.log4j.PatternLayoutlog4j.appender.flume.layout.ConversionPattern=%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n

2，采用xml方式：

<appender name="flume" class="org.apache.flume.clients.log4jappender.LoadBalancingLog4jAppender">  <param name="Hosts" value="10.0.1.75:5555 10.0.1.76:5555 10.0.1.77:5555" /><param name="Selector" value="RANDOM" />        <param name="UnsafeMode" value="true" />        <param name="MaxBackoff" value="30000" /><layout class="org.apache.log4j.PatternLayout"><param name="ConversionPattern" value="%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n" /></layout>  </appender>

测试代码：

protected static final Logger logger = Logger.getLogger(WriteLog.class);@Testpublic void testLog4jAppender() {int i = 1;while (true) {try {logger.error(String.valueOf(new Date().getTime()) + "---第几个：" + i, new NullPointerException("llllll"));i++;Thread.sleep(2000);} catch (Exception e) {e.printStackTrace();}}}

服务端Flume Agent

a1.sources = r1a1.sinks = k1a1.channels = c1# Describe/configure the sourcea1.sources.r1.type = avroa1.sources.r1.bind = 0.0.0.0a1.sources.r1.port = 5555# Describe the sinka1.sinks.k1.type = file_rolla1.sinks.k1.sink.directory = /data/a1.sinks.k1.sink.rollInterval = 0a1.sinks.k1.sink.serializer = TEXTa1.sinks.k1.sink.batchSize = 100# Use a channel which buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# Bind the source and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel = c1

如果是采用LoadBalancingLog4jAppender方式，需要部署多台Flume Agent。配置雷同。