Spark word count 实例

来源:互联网 发布:js event button 编辑:程序博客网 时间:2024/06/06 19:43

利用中午休息时间配置了个环境,写个Word count 本地模式测试下 环境 , 环境Okay

Scala 2.10.4
Java  1.7.08
Spark 1.6.0


package com.ib.e3import org.apache.spark.{SparkConf, SparkContext}/**  * Created by xxxooo on 8/29/2016.  */object WordCount {  def main(args: Array[String]) {    val conf = new SparkConf().setAppName("Myapp").setMaster("local")    val sc = new SparkContext(conf)    val linesRDD = sc.textFile("C:\\workplacebus\\business\\xxxooo\\xxxooo\\workspace\\spark\\lib\\wordcount.txt")    val wordRDD = linesRDD.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey((a,b) => a+b)    wordRDD.foreach(word => println(word))  }}



每一个坚持大汗淋漓的人都不会太简单













0 0