Spark--Scala Actor统计多文件WordCount
来源:互联网 发布:mac mini 发布 编辑:程序博客网 时间:2024/06/05 11:33
import scala.actors.{Future, Actor}import scala.collection.mutableimport scala.collection.mutable.ArrayBufferimport scala.io.Source/** * Created by Administrator on 2017/7/30. */class MyTask extends Actor{ def act(): Unit ={ loop{ react{ case SubmitTask(fileName) =>{ val result = Source.fromFile(fileName).getLines().flatMap(_.split(" ")).map((_,1)).toList.groupBy(_._1).mapValues(_.size) sender ! ResultTask(result) } case StopTask =>{ exit() } } } }}case class SubmitTask(fileName:String)case class ResultTask(result:Map[String,Int])case object StopTaskobject ActorWordCount extends App{ val replySet = new mutable.HashSet[Future[Any]]() val resultList = new ArrayBuffer[ResultTask]() val files = Array[String]("E:\\hadoopSoftWare\\data\\wordCnt.txt","E:\\hadoopSoftWare\\data\\wordCount.txt") for(file <- files){ val task = new MyTask val reply = task.start()!!SubmitTask(file) replySet += reply } while(replySet.size>0){ val toCompute = replySet.filter(_.isSet) for(r <- toCompute){ val result = r.apply().asInstanceOf[ResultTask] //r.apply或者r()取出返回值Future中所谓any的值 resultList += result replySet -= r } } val value = resultList.toList.flatMap(_.result).groupBy(_._1).mapValues(_.foldLeft(0)(_+_._2)); println(value)}
阅读全文
0 0
- Spark--Scala Actor统计多文件WordCount
- 使用scala的Actor模拟计算多文件WordCount
- Scala-Actor并行wordcount
- Spark wordcount - Python, Scala, Java
- Spark+scala+Idea wordcount 示例
- spark(5)--scala之actor模型
- Spark基础-Scala中actor的操作
- 快学Scala-Actor并发编程实现WordCount
- Spark Run WordCount On Hdfs using Scala
- Spark:用Scala和Java实现WordCount
- Spark:用Scala和Java实现WordCount
- Spark:用Scala和Java实现WordCount
- Spark:用Scala和Java实现WordCount
- Spark:用Scala和Java实现WordCount
- Spark:用Scala和Java实现WordCount
- scala-eclipse 编写spark简单程序 WordCount
- Spark:用Scala和Java实现WordCount
- Spark:用Java和Scala实现WordCount
- JSP之自定义标签
- ui-router参数传递问题
- Android----------线性布局和相对布局的使用
- u3d简单的第三人称人物移动与相机跟随
- 十、Java中的类型转换
- Spark--Scala Actor统计多文件WordCount
- linux服务器上配置tomcat以service方式启动
- 文件归档压缩和文件传输
- 编程路上,对于迷失者的一些小小建议
- JS说古道今
- 一张图带你了解游戏程序员的学习之路
- 《Spring中的模板设计模式》
- javaweb--Tomcat>>用自己编译的Tomcat发布应用
- Strust2使用的入门案例__登录案例