一个spark receiver 或多个spark receiver 接收 多个flume agent
来源:互联网 发布:炉石秋日 知乎 编辑:程序博客网 时间:2024/04/29 00:09
- 利用一个spark receiver 接收多个flume agent
String host = args[0]; int port = Integer.parseInt(args[1]); String host1 = args[2]; int port1 = Integer.parseInt(args[3]); InetSocketAddress address1 = new InetSocketAddress(host,port); InetSocketAddress address2 = new InetSocketAddress(host1,port1); InetSocketAddress[] InetSocketAddressArray = {address1,address2};JavaStreamingContext jssc = new JavaStreamingContext(new SparkConf().setAppName("JavaFlumeEventHandle_1"), Durations.seconds(2)); JavaReceiverInputDStream<SparkFlumeEvent> flumeStream = FlumeUtils.createPollingStream(jssc, InetSocketAddressArray, StorageLevel.MEMORY_AND_DISK_SER_2());
- 2个spark receiver接收多个flume agent
String host = args[0]; int port = Integer.parseInt(args[1]); String host1 = args[2]; int port1 = Integer.parseInt(args[3]); InetSocketAddress address1 = new InetSocketAddress(host,port); InetSocketAddress address2 = new InetSocketAddress(host1,port1); InetSocketAddress[] InetSocketAddressArray = {address1,address2}; JavaStreamingContext jssc = new JavaStreamingContext(new SparkConf().setAppName("JavaFlumeEventHandle_1"), Durations.seconds(2));// JavaReceiverInputDStream<SparkFlumeEvent> flumeStream = FlumeUtils.createPollingStream(jssc, InetSocketAddressArray, StorageLevel.MEMORY_AND_DISK_SER_2()); JavaReceiverInputDStream<SparkFlumeEvent> flumeStream = FlumeUtils.createPollingStream(jssc,host,port); JavaReceiverInputDStream<SparkFlumeEvent> flumeStream1 = FlumeUtils.createPollingStream(jssc, host1, port1); JavaDStream<SparkFlumeEvent> union = flumeStream.union(flumeStream1);
性能参照:
http://blog.sina.com.cn/s/blog_13122bdbc0102vay7.html
0 0
- 一个spark receiver 或多个spark receiver 接收 多个flume agent
- 一个spark receiver 或多个spark receiver 接收 多个flume agent
- spark receiver工作内幕
- 10.Spark Streaming源码分析:Receiver数据接收全过程详解
- 10.Spark Streaming源码分析:Receiver数据接收全过程详解
- 【Android】Broadcast Receiver的基本使用,推一条广播给多个Receiver
- Flume多个agent到colletor
- Spark streaming整合Kafka之Receiver方式
- spark submit 多个配置文件
- <receiver>
- Receiver
- Spark如何在一个SparkContext中提交多个任务
- 图解StreamingContext、DStream、Receiver 第三讲spark streaming
- Spark Streaming源码解读之Receiver在Driver详解
- spark streaming 2.02 Receiver 运行executor的调度算法
- Spark Streaming源码解读之Receiver在Driver详解
- spark-streaming-[6]-KafkaWordCount和KafkaWordCountProducer(Receiver-based Approach)
- Spark源码解析之SparkStreaming中Receiver的启动
- Ant_的最完整build.xml解释
- Discuz模版制作流程
- 使用sqlserver 给每组记录顺序编号
- 发布app到app市场
- VS2010中的调试技巧总结
- 一个spark receiver 或多个spark receiver 接收 多个flume agent
- Posix多线程笔记(三)—线程属性(3)
- iBatis 开源框架
- java Dom解析
- Java线程
- QT4.7.3安装mysql5.5系类驱动
- Yii中使用RBAC完全指南
- Codeforces Round #177 (Div. 2)---E. Polo the Penguin and XOR operation(贪心)
- weak-and算法原理演示(wand)