Spark2.x学习笔记：16、Spark Streaming入门实例NetworkWordCount

16、 Spark Streaming入门实例NetworkWordCount

---

16.1 源码解析

在Spark源码的spark-2.1.0-bin-hadoop2.7\examples\src\main\scala\org\apache\spark\examples\streaming目录下即可找到NetworkWordCount.scala源文件。

// scalastyle:off printlnpackage org.apache.spark.examples.streamingimport org.apache.spark.SparkConfimport org.apache.spark.storage.StorageLevelimport org.apache.spark.streaming.{Seconds, StreamingContext}object NetworkWordCount {  def main(args: Array[String]) {    if (args.length < 2) {      System.err.println("Usage: NetworkWordCount <hostname> <port>")      System.exit(1)    }    StreamingExamples.setStreamingLogLevels()    // Create the context with a 1 second batch size    //创建SparkConf实例    val sparkConf = new SparkConf().setAppName("NetworkWordCount")    //每隔1秒钟处理一批数据    val ssc = new StreamingContext(sparkConf, Seconds(1))    // Create a socket stream on target ip:port and count the    // words in input stream of \n delimited text (eg. generated by 'nc')    // Note that no duplication in storage level only for running locally.    // Replication necessary in distributed scenario for fault tolerance.    //创建一个socket流，监听args(0)的args(1)端口输入的数据    val lines = ssc.socketTextStream(args(0), args(1).toInt, StorageLevel.MEMORY_AND_DISK_SER)    //flatMap是把将每一行使用空格做分解，words是个集合，存放单词    val words = lines.flatMap(_.split(" "))    //单词个数统计    val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)    wordCounts.print()    //启动计算作业    ssc.start()    //等待结束，什么时候结束作业，即触发什么条件会让作业执行结束    ssc.awaitTermination()  }}// scalastyle:on println

16.2 测试运行

（1）修改CPU核数
因为我的Spark程序是在虚拟机中运行，如果虚拟机是单核，会导致NetworkWordCount卡住。所以需要修改虚拟机的核心数，这里修改node1节点为2个核即可。

（2）nc
nc是netcat的命令。如果没有安装nc软件，可以通过下面命令安装

[root@node1 ~]# yum install nc

[root@node1 ~]# nc --help...  -l, --listen               Bind and listen for incoming connections  -k, --keep-open            Accept multiple connections in listen mode...

永久监听TCP端口9999

[root@node1 ~]# nc -lk 9999

（3）启动NetworkWordCount

[root@node1 ~]# run-example org.apache.spark.examples.streaming.NetworkWordCount localhost 999917/11/01 09:26:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable-------------------------------------------Time: 1509542803000 ms--------------------------------------------------------------------------------------Time: 1509542804000 ms--------------------------------------------------------------------------------------Time: 1509542805000 ms--------------------------------------------------------------------------------------Time: 1509542806000 ms--------------------------------------------------------------------------------------Time: 1509542807000 ms-------------------------------------------

（4）流处理

在9999端口不停输入数据

[root@node1 ~]# nc -lk 9999hihellohow do you do!i am hadronjavahadoopspark

可以看到NetworkWordCount 实时处理结果

[root@node1 ~]# run-example org.apache.spark.examples.streaming.NetworkWordCount localhost 999917/11/01 09:26:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable-------------------------------------------Time: 1509542803000 ms--------------------------------------------------------------------------------------Time: 1509542804000 ms--------------------------------------------------------------------------------------Time: 1509542805000 ms--------------------------------------------------------------------------------------Time: 1509542806000 ms--------------------------------------------------------------------------------------Time: 1509542807000 ms--------------------------------------------------------------------------------------Time: 1509542808000 ms--------------------------------------------------------------------------------------Time: 1509542809000 ms--------------------------------------------------------------------------------------Time: 1509542810000 ms--------------------------------------------------------------------------------------Time: 1509542811000 ms--------------------------------------------------------------------------------------Time: 1509542812000 ms-------------------------------------------(hi,1)-------------------------------------------Time: 1509542813000 ms--------------------------------------------------------------------------------------Time: 1509542814000 ms--------------------------------------------------------------------------------------Time: 1509542815000 ms-------------------------------------------(hello,1)-------------------------------------------Time: 1509542816000 ms--------------------------------------------------------------------------------------Time: 1509542817000 ms--------------------------------------------------------------------------------------Time: 1509542818000 ms--------------------------------------------------------------------------------------Time: 1509542819000 ms--------------------------------------------------------------------------------------Time: 1509542820000 ms--------------------------------------------------------------------------------------Time: 1509542821000 ms--------------------------------------------------------------------------------------Time: 1509542822000 ms--------------------------------------------------------------------------------------Time: 1509542823000 ms-------------------------------------------(do!,1)(how,1)(you,1)(do,1)-------------------------------------------Time: 1509542824000 ms--------------------------------------------------------------------------------------Time: 1509542825000 ms--------------------------------------------------------------------------------------Time: 1509542826000 ms--------------------------------------------------------------------------------------Time: 1509542827000 ms--------------------------------------------------------------------------------------Time: 1509542828000 ms--------------------------------------------------------------------------------------Time: 1509542829000 ms--------------------------------------------------------------------------------------Time: 1509542830000 ms--------------------------------------------------------------------------------------Time: 1509542831000 ms--------------------------------------------------------------------------------------Time: 1509542832000 ms-------------------------------------------(hadron,1)(am,1)(i,1)-------------------------------------------Time: 1509542833000 ms--------------------------------------------------------------------------------------Time: 1509542834000 ms--------------------------------------------------------------------------------------Time: 1509542835000 ms--------------------------------------------------------------------------------------Time: 1509542836000 ms--------------------------------------------------------------------------------------Time: 1509542837000 ms--------------------------------------------------------------------------------------Time: 1509542838000 ms--------------------------------------------------------------------------------------Time: 1509542839000 ms--------------------------------------------------------------------------------------Time: 1509542840000 ms--------------------------------------------------------------------------------------Time: 1509542841000 ms--------------------------------------------------------------------------------------Time: 1509542842000 ms--------------------------------------------------------------------------------------Time: 1509542843000 ms--------------------------------------------------------------------------------------Time: 1509542844000 ms--------------------------------------------------------------------------------------Time: 1509542845000 ms--------------------------------------------------------------------------------------Time: 1509542846000 ms-------------------------------------------(java,1)-------------------------------------------Time: 1509542847000 ms--------------------------------------------------------------------------------------Time: 1509542848000 ms--------------------------------------------------------------------------------------Time: 1509542849000 ms-------------------------------------------(hadoop,1)-------------------------------------------Time: 1509542850000 ms--------------------------------------------------------------------------------------Time: 1509542851000 ms-------------------------------------------(spark,1)-------------------------------------------Time: 1509542852000 ms