kafka和flume整合

来源：互联网发布：部落冲突劈防空数据编辑：程序博客网时间：2024/05/03 05:36

Kafka作为source：

配置文件：

**#定义各个模块**a1.sources = kafka a1.sinks = loga1.channels = c1#配置kafka source#source的类型为kafkaSourcea1.sources.kafka.type = org.apache.flume.source.kafka.KafkaSource#消费者连接的zk集群地址a1.sources.kafka.zookeeperConnect = crxy155:2181,crxy156:2181,crxy162:2181#消费者消费的topic，只能是一个。a1.sources.kafka.topic = hello#kafka的组ida1.sources.kafka.groupId = flume#kafka的消费者连接超时时间单位毫秒a1.sources.kafka.kafka.consumer.timeout.ms = 3000# 配置logger sinka1.sinks.log.type = logger# 配置 memory channela1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# 绑定三种组件的关系a1.sources.kafka.channels = c1a1.sinks.log.channel = c1

Kafka作为sink：

############### 注意要查看下kafka的lib中的一些依赖包到flume的lib下#####################定义各个模块a1.sources = netcat a1.sinks = kfka1.channels = c1#配置netcat sourcea1.sources.netcat.type = netcata1.sources.netcat.bind = 0.0.0.0a1.sources.netcat.port = 44444# 配置 kafka sinka1.sinks.kfk.type = org.apache.flume.sink.kafka.KafkaSink#topic 如果header里有“topic”字段，会使用header里topic对应的值。a1.sinks.kfk.topic = helloa1.sinks.kfk.brokerList = crxy155:9092,crxy156:9092,crxy162:9092#如果header里含有key这个header则会根据key进行分区。# 配置 memory channela1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactionCapacity = 100# 绑定三种组件的关系a1.sources.netcat.channels = c1a1.sinks.kfk.channel = c1

Kafka作为channel：

**Kafka作为channel：三种方式：
With Flume source and sink
With Flume source and interceptor but no sink
With Flume sink, but no source**

#定义各个模块a1.sources = netcata1.channels = kafka#配置netcat sourcea1.sources.netcat.type = netcata1.sources.netcat.bind = 0.0.0.0a1.sources.netcat.port = 44444# 配置 kafka channela1.channels.kafka.type = org.apache.flume.channel.kafka.KafkaChannela1.channels.kafka.capacity = 10000a1.channels.kafka.transactionCapacity = 1000a1.channels.kafka.zookeeperConnect= crxy155:2181,crxy156:2181,crxy162:2181a1.channels.kafka.brokerList=crxy155:9092,crxy156:9092,crxy162:9092a1.channels.kafka.topic=helloa1.channels.kafka.groupId=flume# 绑定组件的关系a1.sources.netcat.channels = kafka

1 0