flumeng收集日志并存储到HDFS

两台机器agent1收集日志，传送到agent3，agent3上传日志到HDFS

agent1配置：

agent1.sources=s1agent1.channels=c1agent1.sinks=k1agent1.sources.s1.type = spooldiragent1.sources.s1.spoolDir = /var/log/hutxlogagent1.sources.s1.fileHeader =falseagent1.sources.s1.channels = c1agent1.sinks.k1.channel=c1agent1.sinks.k1.type=avroagent1.sinks.k1.hostname=10.1.72.97agent1.sinks.k1.port=12343agent1.channels.c1.type=memoryagent1.sinks.sink-1.type = hdfsagent1.sinks.sink-1.hdfs.kerberosPrincipal=$KERBEROS_PRINCIPALagent1.sinks.sink-1.hdfs.kerberosKeytab =$KERBEROS_KEYTABagent1.sinks.sink-1.hdfs.proxyUser = weblogs

agent3配置

agent3.channels.ch1.type = memoryagent3.sources.avro-source1.channels = ch1agent3.sources.avro-source1.type = avroagent3.sources.avro-source1.bind = 0.0.0.0agent3.sources.avro-source1.port = 12343agent3.sinks.log-sink1.type=hdfsagent3.sinks.log-sink1.channel=ch1agent3.sinks.log-sink1.hdfs.path=hdfs://10.1.72.94:8020/hutxagent3.sinks.log-sink1.hdfs.fileType=DataStreamagent3.sinks.log-sink1.hdfs.writeFormat=TEXTagent3.sinks.log-sink1.hdfs.rollInterval=4agent3.channels = ch1agent3.sources = avro-source1agent3.sinks = log-sink1agent3.sinks.log-sink1.type = hdfsagent3.sinks.log-sink1.hdfs.kerberosPrincipal=$KERBEROS_PRINCIPALagent3.sinks.log-sink1.hdfs.kerberosKeytab =$KERBEROS_KEYTAB

结果

下午3点05:51.301INFOorg.apache.flume.instrumentation.MonitoredCounterGroupShutdown Metric for type: CHANNEL, name: ch1. channel.capacity == 100下午3点05:51.301INFOorg.apache.flume.instrumentation.MonitoredCounterGroupShutdown Metric for type: CHANNEL, name: ch1. channel.current.size == 95下午3点05:51.301INFOorg.apache.flume.instrumentation.MonitoredCounterGroupShutdown Metric for type: CHANNEL, name: ch1. channel.event.put.attempt == 95下午3点05:51.301INFOorg.apache.flume.instrumentation.MonitoredCounterGroupShutdown Metric for type: CHANNEL, name: ch1. channel.event.put.success == 95下午3点05:51.301INFOorg.apache.flume.instrumentation.MonitoredCounterGroupShutdown Metric for type: CHANNEL, name: ch1. channel.event.take.attempt == 47下午3点05:51.301INFOorg.apache.flume.instrumentation.MonitoredCounterGroupShutdown Metric for type: CHANNEL, name: ch1. channel.event.take.success == 0下午3点05:51.301INFOorg.mortbay.logStopped SelectChannelConnector@0.0.0.0:41414下午3点06:03.092INFOorg.apache.flume.node.PollingPropertiesFileConfigurationProviderConfiguration provider starting下午3点06:03.122INFOorg.apache.flume.node.PollingPropertiesFileConfigurationProviderReloading configuration file:/var/run/cloudera-scm-agent/process/1444-flume-AGENT/flume.conf下午3点06:03.127INFOorg.apache.flume.conf.FlumeConfigurationProcessing:log-sink1下午3点06:03.128INFOorg.apache.flume.conf.FlumeConfigurationProcessing:log-sink1下午3点06:03.128INFOorg.apache.flume.conf.FlumeConfigurationProcessing:log-sink1下午3点06:03.129INFOorg.apache.flume.conf.FlumeConfigurationProcessing:log-sink1下午3点06:03.129INFOorg.apache.flume.conf.FlumeConfigurationProcessing:log-sink1下午3点06:03.129INFOorg.apache.flume.conf.FlumeConfigurationAdded sinks: log-sink1 Agent: agent3下午3点06:03.129INFOorg.apache.flume.conf.FlumeConfigurationProcessing:log-sink1下午3点06:03.129INFOorg.apache.flume.conf.FlumeConfigurationProcessing:log-sink1下午3点06:03.129INFOorg.apache.flume.conf.FlumeConfigurationProcessing:log-sink1下午3点06:03.155INFOorg.apache.flume.conf.FlumeConfigurationPost-validation flume configuration contains configuration for agents: [agent3]下午3点06:03.156INFOorg.apache.flume.node.AbstractConfigurationProviderCreating channels下午3点06:03.166INFOorg.apache.flume.channel.DefaultChannelFactoryCreating instance of channel ch1 type memory下午3点06:03.172INFOorg.apache.flume.node.AbstractConfigurationProviderCreated channel ch1下午3点06:03.173INFOorg.apache.flume.source.DefaultSourceFactoryCreating instance of source avro-source1, type avro下午3点06:03.196INFOorg.apache.flume.sink.DefaultSinkFactoryCreating instance of sink: log-sink1, type: hdfs下午3点06:03.786INFOorg.apache.flume.sink.hdfs.HDFSEventSinkHadoop Security enabled: true下午3点06:03.816INFOorg.apache.flume.sink.hdfs.HDFSEventSink{ Sink type:HDFSEventSink, name:log-sink1 }: Attempting kerberos login as principal (flume/gcstat3@CDH) from keytab file (/var/run/cloudera-scm-agent/process/1444-flume-AGENT/flume.keytab)下午3点06:04.107INFOorg.apache.hadoop.security.UserGroupInformationLogin successful for user flume/gcstat3@CDH using keytab file /var/run/cloudera-scm-agent/process/1444-flume-AGENT/flume.keytab下午3点06:04.107INFOorg.apache.flume.sink.hdfs.HDFSEventSinkAuth method: KERBEROS下午3点06:04.107INFOorg.apache.flume.sink.hdfs.HDFSEventSinkUser name: flume/gcstat3@CDH下午3点06:04.107INFOorg.apache.flume.sink.hdfs.HDFSEventSinkUsing keytab: true下午3点06:04.108INFOorg.apache.flume.sink.hdfs.HDFSEventSinkLogged in as user flume/gcstat3@CDH下午3点06:04.112INFOorg.apache.flume.node.AbstractConfigurationProviderChannel ch1 connected to [avro-source1, log-sink1]下午3点06:04.121INFOorg.apache.flume.node.ApplicationStarting new configuration:{ sourceRunners:{avro-source1=EventDrivenSourceRunner: { source:Avro source avro-source1: { bindAddress: 0.0.0.0, port: 12343 } }} sinkRunners:{log-sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@2f93e4a8 counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} }下午3点06:04.125INFOorg.apache.flume.node.ApplicationStarting Channel ch1下午3点06:04.180INFOorg.apache.flume.instrumentation.MonitoredCounterGroupMonitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean.下午3点06:04.180INFOorg.apache.flume.instrumentation.MonitoredCounterGroupComponent type: CHANNEL, name: ch1 started下午3点06:04.181INFOorg.apache.flume.node.ApplicationStarting Sink log-sink1下午3点06:04.182INFOorg.apache.flume.node.ApplicationStarting Source avro-source1下午3点06:04.182INFOorg.apache.flume.source.AvroSourceStarting Avro source avro-source1: { bindAddress: 0.0.0.0, port: 12343 }...下午3点06:04.183INFOorg.apache.flume.instrumentation.MonitoredCounterGroupMonitored counter group for type: SINK, name: log-sink1: Successfully registered new MBean.下午3点06:04.183INFOorg.apache.flume.instrumentation.MonitoredCounterGroupComponent type: SINK, name: log-sink1 started下午3点06:04.238INFOorg.mortbay.logLogging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog下午3点06:04.305INFOorg.mortbay.logjetty-6.1.26下午3点06:04.336INFOorg.mortbay.logStarted SelectChannelConnector@0.0.0.0:41414下午3点06:04.579INFOorg.apache.flume.instrumentation.MonitoredCounterGroupMonitored counter group for type: SOURCE, name: avro-source1: Successfully registered new MBean.下午3点06:04.579INFOorg.apache.flume.instrumentation.MonitoredCounterGroupComponent type: SOURCE, name: avro-source1 started下午3点06:04.580INFOorg.apache.flume.source.AvroSourceAvro source avro-source1 started.下午3点06:33.731INFOorg.apache.avro.ipc.NettyServer[id: 0x03072115, /10.1.72.94:59951 => /10.1.72.97:12343] OPEN下午3点06:33.734INFOorg.apache.avro.ipc.NettyServer[id: 0x03072115, /10.1.72.94:59951 => /10.1.72.97:12343] BOUND: /10.1.72.97:12343下午3点06:33.734INFOorg.apache.avro.ipc.NettyServer[id: 0x03072115, /10.1.72.94:59951 => /10.1.72.97:12343] CONNECTED: /10.1.72.94:59951下午3点06:34.197INFOorg.apache.flume.sink.hdfs.HDFSDataStreamSerializer = TEXT, UseRawLocalFileSystem = false下午3点06:34.311INFOorg.apache.flume.sink.hdfs.BucketWriterCreating hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194198.tmp下午3点06:35.644INFOorg.apache.flume.sink.hdfs.BucketWriterRenaming hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194198.tmp to hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194198下午3点06:35.709INFOorg.apache.flume.sink.hdfs.BucketWriterCreating hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194199.tmp下午3点06:35.826INFOorg.apache.flume.sink.hdfs.BucketWriterRenaming hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194199.tmp to hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194199下午3点06:35.879INFOorg.apache.flume.sink.hdfs.BucketWriterCreating hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194200.tmp下午3点06:35.993INFOorg.apache.flume.sink.hdfs.BucketWriterRenaming hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194200.tmp to hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194200下午3点06:36.040INFOorg.apache.flume.sink.hdfs.BucketWriterCreating hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194201.tmp下午3点06:36.151INFOorg.apache.flume.sink.hdfs.BucketWriterRenaming hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194201.tmp to hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194201下午3点06:36.201INFOorg.apache.flume.sink.hdfs.BucketWriterCreating hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194202.tmp下午3点06:36.301INFOorg.apache.flume.sink.hdfs.BucketWriterRenaming hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194202.tmp to hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194202下午3点06:36.344INFOorg.apache.flume.sink.hdfs.BucketWriterCreating hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194203.tmp下午3点06:36.460INFOorg.apache.flume.sink.hdfs.BucketWriterRenaming hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194203.tmp to hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194203下午3点06:36.505INFOorg.apache.flume.sink.hdfs.BucketWriterCreating hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194204.tmp下午3点06:40.568INFOorg.apache.flume.sink.hdfs.BucketWriterRenaming hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194204.tmp to hdfs://10.1.72.94:8020/hutx/FlumeData.1404803194204下午3点06:40.574INFOorg.apache.flume.sink.hdfs.HDFSEventSinkWriter callback called.