pyspark aggregate函数使用问题（aggregate重写，aggregate中的函数参数限制）

今天在学习spark python 编程中使用aggregate 出现几个需要注意的问题，在这里分享一下！！！

• 需求目标：
  将一个int 类型RDD 中各个分区中的数据中最大数字拿出来，拼接成一个字符串
• 实现代码：

l1 = [1,2,3,4,5,6,7,8,9]rdd1 = sc.parallelize(l1,2)rdd1.aggregate(0,lambda a,b: str(max(a,b)),lambda a,b:a+b)

在spark 环境中上面代码报错：TypeError: unorderable types: int() > str()

从百度得知这个错误是，类型错误，应该传入是int 类型，确传入了str ,一开始的时候一脸懵逼。。。

分析aggregate 源码： def aggregate(self, zeroValue, seqOp, combOp):

        """        Aggregate the elements of each partition, and then the results for all        the partitions, using a given combine functions and a neutral "zero        value."        The functions C{op(t1, t2)} is allowed to modify C{t1} and return it        as its result value to avoid object allocation; however, it should not        modify C{t2}.        The first function (seqOp) can return a different result type, U, than        the type of this RDD. Thus, we need one operation for merging a T into        an U and one operation for merging two U        >>> seqOp = (lambda x, y: (x[0] + y, x[1] + 1))        >>> combOp = (lambda x, y: (x[0] + y[0], x[1] + y[1]))        >>> sc.parallelize([1, 2, 3, 4]).aggregate((0, 0), seqOp, combOp)        (10, 4)        >>> sc.parallelize([]).aggregate((0, 0), seqOp, combOp)        (0, 0)        """        def func(iterator):            acc = zeroValue            for obj in iterator:                acc = seqOp(acc, obj)            yield acc        # collecting result of mapPartitions here ensures that the copy of        # zeroValue provided to each partition is unique from the one provided        # to the final reduce call        vals = self.mapPartitions(func).collect()        return reduce(combOp, vals, zeroValue)

追踪：Traceback 到：acc = seqOp(acc, obj)
seqOp 需要传入两个int 类型数据 ，并将返回值做为下一个传参数，在我的代码中 seqOp = lambda a,b: str(max(a,b)) 返回值为str ,豁然开朗！！1

修改：

l1 = [1,2,3,4,5,6,7,8,9]rdd1 = sc.parallelize(l1,2)rdd1.aggregate(0,max,lambda a,b:str(a)+str(b))

这时又出现了一个问题：
上述代码返回值是的：049 或 094

分析aggregate 的执行过程：
1. reduce of partition 0 will be max(0, 1, 2, 3，4) = 4
2. reduce of partition 1 will be max(0, 5, 6，7，8，9) = 9
3. final reduce across partitions will be ‘0’ + ‘4’ + ‘9’ = ‘094’

问题出在这个初始值，这个初始值不论说什么都会报错(试过：None, ‘’)
都不能得到正确的值‘49’

所以只能重写aggregate 方法：

from functools import reduce class MyRDD(RDD):    def __init__(self):        RDD.__init__(self)    def aggregate(self, seqOp, combOp):        def func(iterator):            acc = 0            for obj in iterator:                acc = seqOp(acc, obj)            yield acc        vals = self.mapPartitions(func).collect()        return reduce(combOp, vals)rdd1.__class__ = MyRDD ##将父类实例转化为子类实例的rdd1.aggregate(max,lambda a,b:str(a)+str(b)) ##结果25

结果：‘49’

终于得到了想要的结果！！！！！