Spark:Scala实现action操作
来源:互联网 发布:阿里云备案管理系统 编辑:程序博客网 时间:2024/06/06 00:44
代码如下
package cn.spark.study.coreimport org.apache.spark.SparkConfimport org.apache.spark.SparkContextobject ActionOperation { def main(args: Array[String]){ //reduce() //collect() //count() take() //saveAsTextFile() //countByKey() //foreach } def reduce(){ val conf =new SparkConf().setAppName("reduce").setMaster("local");val sc = new SparkContext(conf);val numberArray = Array(1,2,3,4,5,6,7,8,9,10);val numbers = sc.parallelize(numberArray, 1);val sum = numbers.reduce(_+_);println(sum) } def collect(){ val conf =new SparkConf().setAppName("reduce").setMaster("local");val sc = new SparkContext(conf);val numberArray = Array(1,2,3,4,5,6,7,8,9,10);val numbers = sc.parallelize(numberArray, 1);val doubleNumbers = numbers.map(num =>num *2)val doubleNumberArray = doubleNumbers.collect()for(num<-doubleNumberArray){ println(num)} } def count(){ val conf =new SparkConf().setAppName("reduce").setMaster("local");val sc = new SparkContext(conf);val numberArray = Array(1,2,3,4,5,6,7,8,9,10);val numbers = sc.parallelize(numberArray, 1);val count = numbers.count();println(count) }def take(){val conf =new SparkConf().setAppName("reduce").setMaster("local");val sc = new SparkContext(conf);val numberArray = Array(1,2,3,4,5,6,7,8,9,10);val numbers = sc.parallelize(numberArray, 1);val top3number = numbers.take(3);println(top3number)// for(num<- top3number){// println(num)// } } def saveAsTextFile(){ val conf =new SparkConf().setAppName("reduce").setMaster("local");val sc = new SparkContext(conf);val numberArray = Array(1,2,3,4,5,6,7,8,9,10);val numbers = sc.parallelize(numberArray, 1);val doubleNumbers = numbers.map(num =>num *2)doubleNumbers.saveAsTextFile("E://BigData//sparkdata//doule_number_scala") } def countByKey(){ val conf = new SparkConf().setAppName("countByKey").setMaster("local"); val sc = new SparkContext(conf); val studentList = Array(Tuple2("class1","leo"),Tuple2("class2","jack"),Tuple2("class1","tom"),Tuple2("class2","mike")) val students = sc.parallelize(studentList, 1) val studentCounts = students.countByKey(); println(studentCounts) }}
阅读全文
0 0
- Spark:Scala实现action操作
- Spark:Java实现Action操作
- Spark中的各种action算子操作(scala版)
- spark连接数据库操作(scala实现)
- Spark:Scala实现常用的Transformation操作
- Spark之Action操作
- Spark action 操作列表
- Spark开发-Action操作
- Spark Action操作
- Spark 的键值对(pair RDD)操作,Scala实现
- spark: scala实现worldcount
- scala spark hbase 操作案例
- Spark Scala DataFram join 操作
- Spark基础-Scala集合操作
- Spark基础-Scala文件操作
- spark action 操作(1)
- Spark:Scala实现KMeans算法
- spark AUC 计算实现(Scala)
- 程序人生
- 线性筛素数
- java.util 时间操作之Date类
- Samba 企业应用案例
- 一步到位的学习网址---岁月中沉淀下来的小经验分享给大家
- Spark:Scala实现action操作
- git笔记
- 递归下降语法分析实验
- Spring在Web中的应用
- Android Studio——布局管理器
- Java字节码4-使用Java-Agent实现一个JVM监控工具
- Android 反编译(三)java文件 和 .smail文件判断和循环比较
- shell中的一些后台执行命令(crontab/at/&/nohup)
- 指尖上行--移动前端开发进阶之路(读书笔记)----2.1动画形式