Spark组件之Spark Streaming学习1--NetworkWordCount学习
来源:互联网 发布:月薪5000如何理财 知乎 编辑:程序博客网 时间:2024/05/16 15:37
更多代码请见:https://github.com/xubo245/SparkLearning
NetworkWordCount:每个1秒将接收的数据进行wordCount,不累加
使用
1.方法1:在集群的examples中启动
一个terminal:
nc -lk 9999
可以在这个terminal发送数据,前面一个terminal就会统计信息另一个terminal:
./bin/run-example streaming.NetworkWordCount localhost 9999
2.运行方法2:打成jar包上传运行:
运行脚本:
#!/usr/bin/env bash spark-submit --name WordCountSpark \--class org.apache.spark.Streaming.learning.NetworkWordCount \--master spark://<strong>Master</strong>:7077 \--executor-memory 512M \--total-executor-cores 10 Streaming.jar localhost 9999
然后一个ternimal运行nc,一个运行这个脚本,同上
输入数据:
hadoop@Master:~$ sudo nc -lk 9999ahelloworldahelloworldhellohw^Hellowordaaaaaaa
结果输出:
hadoop@Master:~/cloud/testByXubo/spark/Streaming$ ./submitJob.sh ------------------------------------------- Time: 1461661853000 ms--------------------------------------------------------------------------------------Time: 1461661854000 ms-------------------------------------------(,1)(hello,1)(world,1)(a,1)-------------------------------------------Time: 1461661855000 ms-------------------------------------------(a,1)-------------------------------------------Time: 1461661856000 ms--------------------------------------------------------------------------------------Time: 1461661857000 ms-------------------------------------------(hello,1)-------------------------------------------Time: 1461661858000 ms-------------------------------------------(world,1)-------------------------------------------Time: 1461661859000 ms--------------------------------------------------------------------------------------Time: 1461661860000 ms-------------------------------------------(hello,1)-------------------------------------------Time: 1461661861000 ms--------------------------------------------------------------------------------------Time: 1461661862000 ms-------------------------------------------(hello,1)-------------------------------------------Time: 1461661863000 ms-------------------------------------------(word,1)------------------------------------------- Time: 1461661864000 ms-------------------------------------------(a,5)------------------------------------------- Time: 1461661865000 ms-------------------------------------------
代码:
/* * Licensed to the Apache Software Foundation (ASF) under one or more * contributor license agreements. See the NOTICE file distributed with * this work for additional information regarding copyright ownership. * The ASF licenses this file to You under the Apache License, Version 2.0 * (the "License"); you may not use this file except in compliance with * the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */// scalastyle:off printlnpackage org.apache.spark.Streaming.learningimport org.apache.spark.SparkConfimport org.apache.spark.storage.StorageLevelimport org.apache.spark.streaming.Secondsimport org.apache.spark.streaming.StreamingContextimport org.apache.spark.streaming.dstream.DStream.toPairDStreamFunctions/** * Counts words in UTF8 encoded, '\n' delimited text received from the network every second. * * Usage: NetworkWordCount <hostname> <port> * <hostname> and <port> describe the TCP server that Spark Streaming would connect to receive data. * * To run this on your local machine, you need to first run a Netcat server * `$ nc -lk 9999` * and then run the example * `$ bin/run-example org.apache.spark.examples.streaming.NetworkWordCount localhost 9999` */object NetworkWordCount { def main(args: Array[String]) { if (args.length < 2) { System.err.println("Usage: NetworkWordCount <hostname> <port>") System.exit(1) } StreamingExamples.setStreamingLogLevels() // Create the context with a 1 second batch size val sparkConf = new SparkConf().setAppName("NetworkWordCount") val ssc = new StreamingContext(sparkConf, Seconds(1)) // Create a socket stream on target ip:port and count the // words in input stream of \n delimited text (eg. generated by 'nc') // Note that no duplication in storage level only for running locally. // Replication necessary in distributed scenario for fault tolerance. val lines = ssc.socketTextStream(args(0), args(1).toInt, StorageLevel.MEMORY_AND_DISK_SER) val words = lines.flatMap(_.split(" ")) val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination() }}// scalastyle:on println
参考:
【1】 http://spark.apache.org/docs/1.5.2/streaming-programming-guide.html
0 0
- Spark组件之Spark Streaming学习1--NetworkWordCount学习
- spark-streaming-[1]-streaming基础NetworkWordCount
- Spark2.x学习笔记:16、Spark Streaming入门实例NetworkWordCount
- Spark组件之Spark Streaming学习2--StatefulNetworkWordCount 学习
- Spark组件之Spark Streaming学习4--HdfsWordCount 学习
- Spark组件之Spark Streaming学习5--WindowsWordCount学习
- [3.1]Spark Streaming初体验之NetworkWordCount案例完美解读
- Spark学习笔记之-Spark-Streaming
- Spark学习之Spark Streaming(9)
- Spark学习之15:Spark Streaming执行流程(1)
- spark streaming的NetworkWordCount实例理解
- spark streaming的NetworkWordCount实例理解
- spark streaming的NetworkWordCount实例理解
- Spark学习笔记-Streaming-1
- Spark Streaming学习(1)
- spark学习笔记:Spark Streaming
- Spark学习六:spark streaming
- Spark组件之Spark Streaming学习3--结合SparkSQL的使用(wordCount)
- lightoj1031 - Easy Game【区间dp】
- 2016随记
- HTMl5的sessionStorage和localStorage
- centos 升级内核到最新版本
- 关于c++包含空格的字符串输入
- Spark组件之Spark Streaming学习1--NetworkWordCount学习
- viewPager 实现banner图 自动轮播
- ios返回上两级页面
- Java IO:CharArrayWriter使用及源码分析
- HDU 4341 - Gold miner
- 距离成为一个优秀程序员 你还差这些
- iOS 贝塞尔曲线和CAShapeLayer 结合使用的进度条详解
- JS中创建函数的三种方式及区别
- moehydrogen.SchoolOlympiadProject