Log4j2的flume appender配置

.由于log4j2比1.x版本的效率据说要高，头疼于日志量太大拖系统资源所以决定试试log4j2，而flume appender就是直接把日志输出到flume agent的方法。
此前使用flume自带的tools——flume-ng-log4jappender，通过在在配置文件中引用其FlumeAppender实现类即可，关于此的文章有很多就不赘述。不过在了解过log4j2后发现，log4j2自带了一个flumeAppender的扩展包，所以自然考虑直接用log4j2提供的flumeAppender插件（事实上你没办法直接用此前的flume-ng-log4jappender，因为log4j2实现appender的结构都已经不同了）。

本文基于的flume版本为1.6.0，log4j 版本为2.3，为叙述方便，以下就不带版本号了。

网上关于log4j2配置flume appender的资料实在太少，log4j2上提供的flume appender的资料不全而且可能有版本滞后之嫌，对现有flume版本是不适用的。当然你可能奇怪为什么要考虑flume的版本，因为log4j提供的FlumeAppender是不带依赖包的，和flume/tools下的flume-ng-log4jappender-1.6.0-jar-with-dependencies.jar可不同，所以你需要大量引入依赖的类，虽然这些包都可以在flume/lib下找到，不过如果你把其下的包都引用的话可能产生一些版本不同的包冲突，所以之后我会把依赖的包详细列出。

一个简单的配置：

......<appenders>        <!--这个输出控制台的配置 -->        <Console name="Console" target="SYSTEM_OUT">            <!--控制台只输出level及以上级别的信息（onMatch），其他的直接拒绝（onMismatch） -->            <ThresholdFilter level="debug" onMatch="ACCEPT" onMismatch="DENY" />            <!--这个都知道是输出日志的格式 -->            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss SSS} %-5level - %-18.18c{2} - %msg%xEx%n" />        </Console>        <!--flume appender的配置，此处采用Avro类型 -->        <Flume name="FlumeAppender" compress="false" type="Avro" >        <Agent host="192.168.2.111" port="4444"/>        <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss SSS} %-5level - %-18.18c{2} - %msg%xEx%n" />      </Flume>    </appenders>    <loggers>        <!--建立一个默认的root的logger -->        <root level="debug">            <appender-ref ref="Console" />        </root>        <!--该logger专为某个类使用，使用了上面定义的FlumeAppender -->        <logger name ="com.ifp.test.testAction" level="debug" additivity="false">        <appender-ref ref="Console" />         <appender-ref ref="FlumeAppender"  />        </logger>    </loggers>    ......

要注意你的flume agent所配置的source只能是avro类型，这无关你appender如何配置。

这个是最容易使用的配置，不过log4j的官方文档中连个example都没有，这次使用log4j似乎比较推荐的embedded类型来实现：

<?xml version="1.0" encoding="UTF-8"?><configuration status="ERROR"><appenders>    <Console name="Console" target="SYSTEM_OUT">            <ThresholdFilter level="debug" onMatch="ACCEPT" onMismatch="DENY" />            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss SSS} %-5level - %-18.18c{2} - %msg%xEx%n" />    </Console>    <Flume name="FlumeAppender" compress="false" type="Embedded" >      <Property name="channel.type">memory</Property>      <Property name="channel.capacity">200</Property>      <Property name="sinks">agent1</Property>      <Property name="agent1.type">avro</Property>      <Property name="agent1.hostname">192.168.0.102</Property>      <Property name="agent1.port">4444</Property>      <Property name="agent1.batch-size">100</Property>      <Property name="processor.type">failover</Property>      <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss SSS} %-5level - %-18.18c{2} - %msg%xEx%n" />      </Flume></appenders><loggers><root level="debug">            <appender-ref ref="Console" />            <appender-ref ref="FlumeAppender" />        </root></loggers></configuration>

乍看你以为是官网给的例子写法（log4j2官网示例），看看是怎么写的——

 <Flume name="eventLogger" compress="true" type="Embedded">      <Property name="channels">file</Property>      <Property name="channels.file.type">file</Property>      <Property name="channels.file.checkpointDir">target/file-channel/checkpoint</Property>      <Property name="channels.file.dataDirs">target/file-channel/data</Property>      <Property name="sinks">agent1 agent2</Property>      <Property name="sinks.agent1.channel">file</Property>      <Property name="sinks.agent1.type">avro</Property>      <Property name="sinks.agent1.hostname">192.168.10.101</Property>      <Property name="sinks.agent1.port">8800</Property>      <Property name="sinks.agent1.batch-size">100</Property>      <Property name="sinks.agent2.channel">file</Property>      <Property name="sinks.agent2.type">avro</Property>      <Property name="sinks.agent2.hostname">192.168.10.102</Property>      <Property name="sinks.agent2.port">8800</Property>      <Property name="sinks.agent2.batch-size">100</Property>      <Property name="sinkgroups">group1</Property>      <Property name="sinkgroups.group1.sinks">agent1 agent2</Property>      <Property name="sinkgroups.group1.processor.type">failover</Property>      <Property name="sinkgroups.group1.processor.priority.agent1">10</Property>      <Property name="sinkgroups.group1.processor.priority.agent2">5</Property>      <RFC5424Layout enterpriseNumber="18060" includeMDC="true" appName="MyApp"/>    </Flume>

这个完全是flume配置文件的格式，但要注意其实这些配置是为了实例化EmbeddedAgent类的配置信息，EmbeddedAgent其实是一个简化的agent，它的source只能是embedded，什么意思呢，其实就是只能通过put(event)或者putAll(events)方法直接把flume Event直接传入agent的channel里去。而EmbeddedAgent的sink只能是avro类型，channel只能是memory或file，而且必须配置sink processor类型。我们可以来看看flume官网关于EmbeddedAgentd的Demo：

Map<String, String> properties = new HashMap<String, String>();properties.put("channel.type", "memory");//必填属性properties.put("channel.capacity", "200");properties.put("sinks", "sink1 sink2");//必填属性properties.put("sink1.type", "avro");//必填属性properties.put("sink2.type", "avro");properties.put("sink1.hostname", "collector1.apache.org");properties.put("sink1.port", "5564");properties.put("sink2.hostname", "collector2.apache.org");properties.put("sink2.port",  "5565");properties.put("processor.type", "load_balance");//必填属性properties.put("source.interceptors", "i1");properties.put("source.interceptors.i1.type", "static");properties.put("source.interceptors.i1.key", "key1");properties.put("source.interceptors.i1.value", "value1");EmbeddedAgent agent = new EmbeddedAgent("myagent");agent.configure(properties);agent.start();List<Event> events = Lists.newArrayList();events.add(event);events.add(event);events.add(event);events.add(event);agent.putAll(events);...agent.stop();

你会发现这和log4j给的属性名完全不同，我猜这可能是flume1.6.0的EmbeddedAgent的改动，所以我之前用官网的配置完全失败了，包括你使用默认配置也会出现问题，估计log4j对flume appender有阵子没有维护了。

最后附上依赖列表，仅供参考：

<dependencies>    <dependency>      <groupId>org.apache.avro</groupId>      <artifactId>avro</artifactId>      <version>1.7.4</version>    </dependency>    <dependency>      <groupId>org.apache.avro</groupId>      <artifactId>avro-ipc</artifactId>      <version>1.7.4</version>    </dependency>    <dependency>      <groupId>org.apache.logging.log4j</groupId>      <artifactId>log4j-core</artifactId>      <version>2.3</version>    </dependency>    <dependency>      <groupId>org.apache.logging.log4j</groupId>      <artifactId>log4j-api</artifactId>      <version>2.3</version>    </dependency>    <dependency>      <groupId>org.apache.logging.log4j</groupId>      <artifactId>log4j-flume-ng</artifactId>      <version>2.3</version>    </dependency>   <dependency>      <groupId>org.apache.logging.log4j</groupId>      <artifactId>log4j-1.2-api</artifactId>      <version>2.3</version>    </dependency>     <dependency>      <groupId>commons-lang</groupId>      <artifactId>commons-lang</artifactId>      <version>2.5</version>    </dependency>    <dependency>     <groupId>commons-logging</groupId>      <artifactId>commons-logging</artifactId>      <version>1.1.1</version>    </dependency>    <dependency>     <groupId>org.slf4j</groupId>      <artifactId>slf4j-log4j12</artifactId>      <version>1.6.1</version>    </dependency>    <dependency>     <groupId>org.slf4j</groupId>      <artifactId>slf4j-api</artifactId>      <version>1.6.1</version>    </dependency>    <dependency>     <groupId>org.apache.flume</groupId>      <artifactId>flume-ng-sdk</artifactId>      <version>1.6.0</version>    </dependency>    <dependency>     <groupId>org.apache.flume</groupId>      <artifactId>flume-ng-embedded-agent</artifactId>      <version>1.6.0</version>    </dependency>   <dependency>     <groupId>org.apache.flume</groupId>      <artifactId>flume-ng-core</artifactId>      <version>1.6.0</version>    </dependency>    <dependency>     <groupId>org.apache.flume</groupId>      <artifactId>flume-ng-configuration</artifactId>      <version>1.6.0</version>    </dependency>    <dependency>     <groupId>org.apache.flume</groupId>      <artifactId>flume-ng-node</artifactId>      <version>1.6.0</version>    </dependency>    <dependency>     <groupId>org.apache.flume.flume-ng-channels</groupId>      <artifactId>flume-file-channel</artifactId>      <version>1.6.0</version>    </dependency>    <dependency>     <groupId>io.netty</groupId>      <artifactId>netty</artifactId>      <!--注意不支持netty4以上-->      <version>3.5.12.Final</version>    </dependency>    <dependency>    <groupId>org.codehaus.jackson</groupId>      <artifactId>jackson-core-asl</artifactId>      <version>1.9.3</version>    </dependency>    <dependency>    <groupId>org.codehaus.jackson</groupId>      <artifactId>jackson-mapper-asl</artifactId>      <version>1.9.3</version>    </dependency>    <dependency>      <groupId>com.google.guava</groupId>      <artifactId>guava</artifactId>      <version>11.0.2</version>    </dependency>  </dependencies>