scala-wordcount

来源：互联网发布：ubuntu怎么安装qq2017 编辑：程序博客网时间：2024/06/04 19:15

/*
* Mapreduce步骤
* 1、map：获取一行
* 2、按空格分隔
* 3、每个单词统计，<key,1> 输出到 reducer
* 4、reducer <key,{1,1,1,1,1}> 汇总
* 5、输出结果
*/

1、声明一个集合，模仿要统计的文档内容，一个逗号分隔的集合元素相当于一行

scala> val lines = List("hello tom hello jerry", "hello jerry", "hello kitty","beijing shanghai tianjing","xiaoming xiaohua xiaoli xiaozhang")

2、 map：获取一行，按空格分隔

scala> val line2=lines.flatMap(_.split(" "))
line2: List[String] = List(hello, tom, hello, jerry, hello, jerry, hello, kitty, beijing, shanghai, tianjing, xiaoming, xiaohua, xiaoli, xiaozhang)

3、每个单词统计，<key,1> 输出到 reducer

scala> val line3=line2.map((_,1))
line3: List[(String, Int)] = List((hello,1), (tom,1), (hello,1), (jerry,1), (hello,1), (jerry,1), (hello,1), (kitty,1), (beijing,1), (shanghai,1), (tianjing,1), (xiaoming,1), (xiaohua,1), (xiaoli,1), (xiaozhang,1))

4、reducer <key,{1,1,1,1,1}> 汇总

scala> val line4=line3.groupBy(_._1)
line4: scala.collection.immutable.Map[String,List[(String, Int)]] = Map(kitty -> List((kitty,1)), xiaoming -> List((xiaoming,1)), tianjing -> List((tianjing,1)), shanghai -> List((shanghai,1)), xiaohua -> List((xiaohua,1)), tom -> List((tom,1)), beijing -> List((beijing,1)), hello -> List((hello,1), (hello,1), (hello,1), (hello,1)), jerry -> List((jerry,1), (jerry,1)), xiaozhang -> List((xiaozhang,1)), xiaoli -> List((xiaoli,1)))

scala> val line5=line4.map(t=>(t._1,t._2.size))
line5: scala.collection.immutable.Map[String,Int] = Map(kitty -> 1, xiaoming -> 1, tianjing -> 1, shanghai -> 1, xiaohua -> 1, tom -> 1, beijing -> 1, hello -> 4, jerry -> 2, xiaozhang -> 1, xiaoli -> 1)

5、输出结果

scala> val line6=line5.toList
line6: List[(String, Int)] = List((kitty,1), (xiaoming,1), (tianjing,1), (shanghai,1), (xiaohua,1), (tom,1), (beijing,1), (hello,4), (jerry,2), (xiaozhang,1), (xiaoli,1))

scala> val line7=line6.sortBy(_._2).reverse
line7: List[(String, Int)] = List((hello,4), (jerry,2), (xiaoli,1), (xiaozhang,1), (beijing,1), (tom,1), (xiaohua,1), (shanghai,1), (tianjing,1), (xiaoming,1), (kitty,1))

-------------

更多的Java，Android，大数据，J2EE，Python，数据库，Linux，Java架构师，教程，视频请访问：

http://www.cnblogs.com/zengmiaogen/p/7083694.html

0 0