【云星数据---Apache Flink实战系列(精品版)】：Apache Flink批处理API详解与编程实战005--DateSet实用API详解005

DateSet的API详解五

groupBy

def groupBy(firstField: String, otherFields: String*): GroupedDataSet[T]def groupBy(fields: Int*): GroupedDataSet[T]def groupBy[K](fun: (T) ⇒ K)(implicit arg0: TypeInformation[K]): GroupedDataSet[T]Creates a GroupedDataSet which provides operations on groups of elements.暗示第二个输入较小的交叉。拿第一个输入的每一个元素和第二个输入的每一个元素进行交叉操作。

groupBy示例一：使用一个Case Class Fields

执行程序：

//1.定义 classcase class WC(val word: String, val salary: Int) //2.定义DataSet[WC]val words: DataSet[WC] = benv.fromElements(WC("LISI",600),WC("LISI",400),WC("WANGWU",300),WC("ZHAOLIU",700))//3.使用自定义的reduce方法,使用key-expressions val wordCounts1 = words.groupBy("word").reduce {    (w1, w2) => new WC(w1.word, w1.salary + w2.salary)}//4.使用自定义的reduce方法,使用key-selectorval wordCounts2 = words.groupBy { _.word } reduce {     (w1, w2) => new WC(w1.word, w1.salary + w2.salary)}//5.显示结果wordCounts1.collectwordCounts2.collect

执行结果：

Scala-Flink> wordCounts1.collectres5: Seq[WC] = Buffer(WC(LISI,1000), WC(WANGWU,300), WC(ZHAOLIU,700))Scala-Flink> wordCounts1.collec2res6: Seq[WC] = Buffer(WC(LISI,1000), WC(WANGWU,300), WC(ZHAOLIU,700))

web ui中的执行效果：

groupBy示例二：使用多个Case Class Fields

执行程序：

//1.定义 case classcase class Student(val name: String, addr: String, salary: Double)//2.定义DataSet[Student]val tuples:DataSet[Student] = benv.fromElements(Student("lisi","shandong",2400.00),Student("zhangsan","henan",2600.00),Student("lisi","shandong",2700.00),Student("lisi","guangdong",2800.00))//3.使用自定义的reduce方法,使用多个Case Class Fields nameval reducedTuples1 = tuples.groupBy("name", "addr").reduce {  (s1, s2) => Student(s1.name+"-"+s2.name,s1.addr+"-"+s2.addr,s1.salary+s2.salary)}//4.使用自定义的reduce方法,使用多个Case Class Fields indexval reducedTuples2 = tuples.groupBy(0, 1).reduce {  (s1, s2) => Student(s1.name+"-"+s2.name,s1.addr+"-"+s2.addr,s1.salary+s2.salary)}//5.使用自定义的reduce方法,name和index混用val reducedTuples3 = tuples.groupBy(0, 1).reduce {  (s1, s2) => Student(s1.name+"-"+s2.name,s1.addr+"-"+s2.addr,s1.salary+s2.salary)}//6.显示结果reducedTuples1.collectreducedTuples2.collectreducedTuples3.collect

执行结果：

Scala-Flink> reducedTuples1.collectres96: Seq[Student] = Buffer(Student(lisi,guangdong,2800.0),Student(lisi-lisi,shandong-shandong,5100.0), Student(zhangsan,henan,2600.0))Scala-Flink> reducedTuples2.collectres97: Seq[Student] = Buffer(Student(lisi,guangdong,2800.0),Student(lisi-lisi,shandong-shandong,5100.0), Student(zhangsan,henan,2600.0))Scala-Flink> reducedTuples3.collectres98: Seq[Student] = Buffer(Student(lisi,guangdong,2800.0),Student(lisi-lisi,shandong-shandong,5100.0), Student(zhangsan,henan,2600.0))

web ui中的执行效果：