Spark Streaming---WindowBasedTopWord(java)
来源:互联网 发布:mac如何拷贝文件到u盘 编辑:程序博客网 时间:2024/05/29 07:49
package com.spark.streaming;import org.apache.spark.SparkConf;import org.apache.spark.api.java.function.Function;import org.apache.spark.api.java.function.Function2;import org.apache.spark.api.java.function.PairFunction;import org.apache.spark.streaming.Durations;import org.apache.spark.streaming.api.java.JavaDStream;import org.apache.spark.streaming.api.java.JavaPairDStream;import org.apache.spark.streaming.api.java.JavaReceiverInputDStream;import org.apache.spark.streaming.api.java.JavaStreamingContext;import scala.Tuple2;public class WindowBasedTopWord { public static void main(String[] args) { SparkConf conf = new SparkConf().setAppName("WindowBasedTopWOrd").setMaster("local[2]") .set("spark.default.parallelism", "100"); JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(5)); // 这里叫log日志, yasaka hello , xuruyun world JavaReceiverInputDStream<String> log = jssc.socketTextStream("node15", 8888); JavaDStream<String> searchWord = log.map(new Function<String, String>() { private static final long serialVersionUID = 1L; @Override public String call(String searchLog) throws Exception { return searchLog.split(" ")[1]; } }); JavaPairDStream<String, Integer> pairs = searchWord.mapToPair(new PairFunction<String, String, Integer>() { private static final long serialVersionUID = 1L; @Override public Tuple2<String, Integer> call(String word) throws Exception { return new Tuple2<String, Integer>(word ,1); } }); JavaPairDStream<String, Integer> wordcounts = pairs.reduceByKeyAndWindow( new Function2<Integer, Integer, Integer>() { private static final long serialVersionUID = 1L; @Override public Integer call(Integer v1, Integer v2) throws Exception { return v1 + v2; } },Durations.seconds(60), Durations.seconds(10)); wordcounts.print(); jssc.start(); jssc.awaitTermination(); jssc.close(); }}
阅读全文
1 0
- Spark Streaming---WindowBasedTopWord(java)
- Spark Streaming---UpdatestateBykey(java)
- Spark Streaming Java案例
- Spark Streaming---Streaming Word Count(java)
- Spark Streaming----Transform Operation(java)
- Spark Streaming
- spark streaming
- Spark/Streaming
- Spark Streaming
- spark streaming
- Spark Streaming
- Spark Streaming
- Spark Streaming
- Spark Streaming
- spark streaming
- Spark Streaming
- Spark Streaming
- Spark Streaming
- CentOS7
- Spark Streaming---UpdatestateBykey(java)
- Android逆向之旅---破解一款永久免费网络访问工具
- 【unique函数】
- codeblocks调试
- Spark Streaming---WindowBasedTopWord(java)
- 数据科学风云之互联网金融
- python实现自动监测目标网站的爬取速度_以及整体网络环境分析
- Redhat7下安装oracle11g
- 每日积累(20170819-day-25)(Ubuntu Nvidia Driver loop in sign)
- 从0开始写一个基于注解的轻量级分布式RPC框架(4)自定义Spring的IOC,自定义属性注入bean的过程
- 跨域问题解决(一):tamcat下的配置
- wamp安装配置问题
- Git十周岁之际,Linus Torvalds大谈Git开发故事