spark2.0原理分析--RDD Lineage（逻辑执行计划）

RDD Lineage（又称为RDD运算图或RDD依赖关系图）是RDD所有父RDD的graph（图）。它是在RDD上执行transformations函数并创建logical execution plan（逻辑执行计划）的结果。它是RDD的逻辑执行计划。
注意： execution DAG或physical execution plan（物理执行计划）是DAG of stages（stage的DAG）。

上图是执行以下语句得到的RDD Lineage结果：

val r00 = sc.parallelize(0 to 9)val r01 = sc.parallelize(0 to 90 by 10)val r10 = r00 cartesian r01val r11 = r00.map(n => (n, n))val r12 = r00 zip r01val r13 = r01.keyBy(_ / 20)val r20 = Seq(r11, r12, r13).foldLeft(r10)(_ union _)

我们可以执行toDebugString打印RDD的Lineage：

scala> r00.toDebugStringres5: String = (20) ParallelCollectionRDD[0] at parallelize at <console>:27 []scala> r01.toDebugStringres6: String = (20) ParallelCollectionRDD[1] at parallelize at <console>:27 []    scala> r12.toDebugStringres9: String = (20) ZippedPartitionsRDD2[4] at zip at <console>:31 [] | ParallelCollectionRDD[0] at parallelize at <console>:27 [] | ParallelCollectionRDD[1] at parallelize at <console>:27 []scala> r13.toDebugStringres10: String = (20) MapPartitionsRDD[5] at keyBy at <console>:29 [] | ParallelCollectionRDD[1] at parallelize at <console>:27 []scala> r20.toDebugStringres11: String = (460) UnionRDD[8] at union at <console>:39 []  | UnionRDD[7] at union at <console>:39 []  | UnionRDD[6] at union at <console>:39 []  | CartesianRDD[2] at cartesian at <console>:31 []  | ParallelCollectionRDD[0] at parallelize at <console>:27 []  | ParallelCollectionRDD[1] at parallelize at <console>:27 []  | MapPartitionsRDD[3] at map at <console>:29 []  | ParallelCollectionRDD[0] at parallelize at <console>:27 []  | ZippedPartitionsRDD2[4] at zip at <console>:31 []  | ParallelCollectionRDD[0] at parallelize at <console>:27 []  | ParallelCollectionRDD[1] at parallelize at <console>:27 []  | MapPartitionsRDD[5] at keyBy at <console>:29 []  | ParallelCollectionRDD[1] at parallelize at <console>:27 []

从以上输出可以看出，RDD谱系图是在调用action函数后需要执行哪些transformation（变换）的graph(图)。

toDebugString

函数原型：

def toDebugString: String

通过此函数可以获取RDD的Lineage打印输出。

设置打印Lineage

参数：spark.logLineage默认值：false设置为true时，会在执行中打印出RDD的Lineage。

Logical Execution Plan（逻辑执行计划）

logical execution plan（逻辑执行计划）从最早（头结点-没有父结点的节点）的RDD（不依赖于其他RDD或引用缓存数据的RDD）开始，并以生成被执行的action操作结果的RDD结束。