Spark mapPartitions()操作
来源:互联网 发布:php小偷源码 编辑:程序博客网 时间:2024/06/07 06:54
原文地址:http://apachesparkbook.blogspot.com/2015/11/mappartition-example.html
---
mapPartitions() can be used as an alternative to map() & foreach(). mapPartitions() is called once for each Partition unlike map() & foreach()which is called for each element in the RDD. The main advantage being that, we can do initialization on Per-Partition basis instead of per-element basis(as done by map() & foreach())
Consider the case of Initializing a database. If we are using map() or foreach(), the number of times we would need to initialize will be equal to the no of elements in RDD. Whereas if we use mapPartitions(), the no of times we would need to initialize would be equal to number of Partitions
We get Iterator as an argument for mapPartition, through which we can iterate through all the elements in a Partition.
In this example, we will use mapPartitionsWithIndex(), which apart from similar to mapPartitions() also provides an index to track the Partition No
Syntax
Syntax
def mapPartitionsWithIndex[U](f: (Int, Iterator[T]) ⇒ Iterator[U], preservesPartitioning: Boolean = false)(implicit arg0: ClassTag[U]): RDD[U]Return a new RDD by applying a function to each partition of this RDD, while tracking the index of the original partition.preservesPartitioning indicates whether the input function preserves the partitioner, which should be false unless this is a pair RDD and the input function doesn't modify the keys.
Example
scala> val rdd1 = sc.parallelize( | List( | "yellow", "red", | "blue", "cyan", | "black" | ), | 3 | )rdd1: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[10] at parallelize at :21scala>scala> val mapped = rdd1.mapPartitionsWithIndex{ | // 'index' represents the Partition No | // 'iterator' to iterate through all elements | // in the partition | (index, iterator) => { | println("Called in Partition -> " + index) | val myList = iterator.toList | // In a normal user case, we will do the | // the initialization(ex : initializing database) | // before iterating through each element | myList.map(x => x + " -> " + index).iterator | } | }mapped: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[11] at mapPartitionsWithIndex at :23scala> | mapped.collect()Called in Partition -> 1Called in Partition -> 2Called in Partition -> 0res7: Array[String] = Array(yellow -> 0, red -> 1, blue -> 1, cyan -> 2, black -> 2)
0 0
- Spark mapPartitions()操作
- Spark算子:RDD基本转换操作(5)–mapPartitions、
- Spark算子:RDD基本转换操作(mapPartitions、mapPartitionsWithIndex)
- Spark算子:RDD基本转换操作(5)–mapPartitions、mapPartitionsWithIndex
- Spark算子:RDD基本转换操作(5)–mapPartitions、mapPartitionsWithIndex
- Spark算子:RDD基本转换操作(5)–mapPartitions/mapPartitionsWithIndex
- Spark算子:RDD基本转换操作(5)–mapPartitions、mapPartitionsWithIndex
- Spark算子:RDD基本转换操作(5)–mapPartitions、mapPartitionsWithIndex
- 3.2 Spark RDD 基本转换操作5-mapPartitions、mapPartitionsWithIndex
- spark:map mapPartitions flatmap
- spark:map mapPartitions flatmap
- Spark中的mapPartitions
- Spark中mapPartitions使用
- spark--transform算子--mapPartitions
- [spark]Spark算子:RDD基本转换操作(5)–mapPartitions、mapPartitionsWithIndex
- spark小技巧-mapPartitions
- Spark Transformation —— mapPartitions
- Spark API 之 map、mapPartitions
- as 获得sha1
- 七种排序算法
- 魅族pro5手机android studio 无法输出Log
- NVidia TensorRT 运行 Caffe 模型
- (36) 默认实参、内联函数和constexpr函数
- Spark mapPartitions()操作
- 平台立体监控nodejs+mongodb
- 单片机应用系统的基本组成
- 一个封装android开发常用控件的库
- Dubbo2.3.2的服务停止脚本stop.sh源码
- js原生代码实现数据双向绑定
- 堆和栈的区别(内存和数据结构)
- Leetcode 98. Validate Binary Search Tree (Medium) (cpp)
- 牛客网刷题之数据流中的中位数