Optimize map performamce with mapPartitions

来源：互联网发布：哪个漫画软件日漫最全编辑：程序博客网时间：2024/05/16 12:43

As we can see in previous article "CSV Parser" we may need to create a new object for each record of an RDD as in

1
2
3
4
5
6
    defmLine(line:String)={
      valparser=newCSVParser('\t')
      parser.parseLine(line)
    }
...
    ...myRDD.map(mLine(_).size)...

The mLine function is used in the map method of an RDD. In this case the parser object is created each time for each record, although they are exactly the same thing.

Actually, whenever we need to apply some complicated operation on each record there is a high chance we need to create some helper objects within map. By combining mapPartition with Scala map, we can reduce the unnecessary new object creation. Let’s rewrite above example with mapPartitions:

1
2
3
4
5
6
    defpLines(lines:Iterator[String])={
      valparser=newCSVParser('\t')
      lines.map(parser.parseLine(_).size)
    }
...
    myRDD.mapPartitions(pLines)

On my single box test machine, execution time of the same task reduced from 65 seconds to 35 seconds. Surprisingly the opencsv parser with the mapPartitions optimization is significantly faster than map(_split('\t')).

0 0