flume+kalka 搭建一
来源:互联网 发布:学生宿舍管理系统源码 编辑:程序博客网 时间:2024/05/24 06:09
flume+kafka搭建
一. kafka集群搭建启动
详细参见文档《kafka集群搭建文档.docx》
安装启动kafka集群
二. 下载安装flume
flume官方下载地址:https://flume.apache.org/download.html
建议下载最新的1.6.0版本的,因为1.6.0版本的集成了整合kafka的插件包可以直接配置使用
下载apache-flume-1.6.0-bin.tar.gz包
通过tar –zxvf apache-flume-1.6.0-bin.tar.gz命令解压
Flume解压既安装成功,配置conf/ flume-conf.properties文件启动完成相应的功能
三. 配置flume连接kafka
前面kafka集群已经成功,这里只需要配置好conf/ flume-conf.properties文件,配置如下
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type=avro
a1.sources.r1.bind=master
a1.sources.r1.port=41414
# Describe the sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic = testflume
a1.sinks.k1.brokerList = 192.168.57.4:9092,192.168.57.5:9092,192.168.57.6:9092
a1.sinks.k1.requiredAcks = 1
a1.sinks.k1.batchSize = 20
a1.sinks.k1.channel = c1
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000000
a1.channels.c1.transactionCapacity = 10000
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
注意这里需要根据实际需求来配置sources,这里是用avro的方式配置监听master本机的41414端口注意这里是启动agent的配置 后续的flume client也需要用到这个配置,下面的sink配置a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink是固定的不需要修改,而a1.sinks.k1.topic = testflume是创建的话题名,需要根据自己需要来改,而a1.sinks.k1.brokerList = 192.168.57.4:9092,192.168.57.5:9092,192.168.57.6:9092是根据实际的kafka集群情况配置的
四.测试
上面配置完成以后启动flume连接到kafka
bin/flume-ng agent -c ./conf/ -f conf/flume-conf.properties -Dflume.root.logger=DEBUG,console -na1
注意这里的a1指的是配置文件中的agent名字a1不是随意写的
这里的flume对接kafka实际已经完成,接下来就是测试
测试代码:
import org.apache.flume.Event;
import org.apache.flume.EventDeliveryException;
import org.apache.flume.api.RpcClient;
import org.apache.flume.api.RpcClientFactory;
import org.apache.flume.event.EventBuilder;
import java.nio.charset.Charset;
public class MyApp {
public static void main(String[] args) {
MyRpcClientFacade client = new MyRpcClientFacade();
// Initialize client with the remote Flume agent's host and port
client.init("master",41414);
// Send 10 events to the remote Flume agent. That agent should be
// configured to listen with an AvroSource.
String sampleData = "Hello Flume!";
for (inti = 0; i < 10; i++) {
client.sendDataToFlume(sampleData);
}
client.cleanUp();
}
}
class MyRpcClientFacade {
private RpcClient client;
private String hostname;
private int port;
public void init(String hostname,int port) {
// Setup the RPC connection
this.hostname= hostname;
this.port= port;
this.client= RpcClientFactory.getDefaultInstance(hostname, port);
// Use the following method to create a thrift client (instead of the above line):
// this.client = RpcClientFactory.getThriftInstance(hostname, port);
}
public void sendDataToFlume(String data) {
// Create a Flume Event object that encapsulates the sample data
Event event = EventBuilder.withBody(data, Charset.forName("UTF-8"));
// Send the event
try{
client.append(event);
} catch (EventDeliveryException e) {
// clean up and recreate the client
client.close();
client = null;
client = RpcClientFactory.getDefaultInstance(hostname,port);
// Use the following method to create a thrift client (instead of the above line):
// this.client = RpcClientFactory.getThriftInstance(hostname, port);
}
}
public void cleanUp() {
// Close the RPC connection
client.close();
}
}
注意这里的client.init("master",41414);中的master和41414是和flume配置文件里保持一致的
先启动一个consumer来监听
./bin/kafka-console-consumer.sh --zookeeper master:2181,slave1:2181,slave2:2181 --from-beginning --topic testflume
注意这里的topic的名字要和配置文件中一致
执行上面的main方法作为flume的client端来产生数据,可以在上面的consumer监听里面看到结果
- flume+kalka 搭建一
- Flume(一) Flume初识 环境的搭建和配置
- 日志系统搭建一(flume+hadoop+hive)
- Flume搭建
- flume搭建
- flume搭建
- Hadoop+Flume+Kafka+Zookeeper集群环境搭建(一)
- Flume教程(一) Flume入门教程
- flume学习一:flume基础知识
- flume搭建调试
- flume搭建调试
- flume开发环境搭建
- Flume集群搭建
- flume集群搭建
- Flume集群搭建
- 搭建flume集群
- Flume搭建测试
- flume开发环境搭建
- Debug日志:流对象更名+流运算符重载实验
- MessageFormat 在sql中批量操作的妙用
- vmware 安装ghost 系统
- TensorFlow框架(2)之TensorBoard详解
- CentOS 7搭建 Node.js 环境
- flume+kalka 搭建一
- struts2-struts2的基本配置
- [方便查看筆記] centos7安裝谷歌瀏覽器
- 微信公众号开发 redirect_uri 参数错误
- 一步一步在Windows中使用MyCat负载均衡
- Principle for Mac 3.2 简体中文完美汉化包 原型设计工具
- java 字节流与字符流的区别
- 面对安逸、焦虑,我选择:自信——拼搏
- C++类虚函数内存分布(非常重要!!!)