flume-kafka整合--实时日志采集

flume采用架构

exec-source + memory-channel + avro-sink

avro-source + memory-channel + kafka-sink

kafka采用架构

启动zookeeper

zkServer.sh start

启动kafka(启动一个“篮子”)

kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties &

模拟

启动一个kafka消费者监听日志文件

kafka-console-consumer.sh –zookeeper 192.168.145.128:2181 –from-beginning –topic firstTopic

向日志文件中添加数据

echo hi,flume-kafka framework >> flume-kafka.txt

---

---

---

exec-memory-avro.conf

flume-ng agent \  --name exec-memory-avro \  --conf $FLUME_HOME/conf \  --conf-file $FLUME_HOME/conf/exec-memory-avro.conf \  -Dflume.root.logger=INFO,console # example exec-memory-avroexec-memory-avro.sources = exec-sourceexec-memory-avro.sinks = avro-sinkexec-memory-avro.channels = memory-channel# Describe/configure the sourceexec-memory-avro.sources.exec-source.type = execexec-memory-avro.sources.exec-source.command = tail -F  /root/data/flume-kafka.txtexec-memory-avro.sources.exec-source.shell = /bin/sh -c# Describe/ the sinkexec-memory-avro.sinks.avro-sink.type = avroexec-memory-avro.sinks.avro-sink.hostname = 192.168.145.128exec-memory-avro.sinks.avro-sink.port = 44444 # Use a channel which buffers events in memoryexec-memory-avro.channels.memory-channel.type = memory# Bind the source and sink to the channelexec-memory-avro.sources.exec-source.channels = memory-channelexec-memory-avro.sinks.avro-sink.channel = memory-channel

avro-memory-kafka.conf

flume-ng agent \  --name avro-memory-kafka \  --conf $FLUME_HOME/conf \  --conf-file $FLUME_HOME/conf/avro-memory-kafka.conf \  -Dflume.root.logger=INFO,console # example avro-memory-kafkaavro-memory-kafka.sources = avro-sourceavro-memory-kafka.sinks = kafka-sinkavro-memory-kafka.channels = memory-channel# Describe/configure the sourceavro-memory-kafka.sources.avro-source.type = avroavro-memory-kafka.sources.avro-source.bind = 192.168.145.128avro-memory-kafka.sources.avro-source.port = 44444# Describe/ the sinkavro-memory-kafka.sinks.kafka-sink.type = org.apache.flume.sink.kafka.KafkaSinkavro-memory-kafka.sinks.kafka-sink.brokerList = 192.168.145.128:9092avro-memory-kafka.sinks.kafka-sink.topic = firstTopicavro-memory-kafka.sinks.kafka-sink.batchSize = 3avro-memory-kafka.sinks.kafka-sink.requiredAcks = 1# Use a channel which buffers events in memoryavro-memory-kafka.channels.memory-channel.type = memory# Bind the source and sink to the channelavro-memory-kafka.sources.avro-source.channels = memory-channelavro-memory-kafka.sinks.kafka-sink.channel = memory-channel