为什么说cache是persist的特例？

有人经常会看到Spark中有句话说：cache是persist的特例。

通过分析源码，我们来看一下，这句话的含义：

\spark-1.5.0\core\src\main\scala\org\apache\spark\rdd\RDD.scala

/** Persist thisRDD with the default storage level (`MEMORY_ONLY`). */

  def persist(): this.type =persist(StorageLevel.MEMORY_ONLY)

 

  /** Persist this RDD with the default storagelevel (`MEMORY_ONLY`). */

  def cache(): this.type = persist()

 

  /**

   * Mark the RDD as non-persistent, and removeall blocks for it from memory and disk.

   * @param blocking Whether to block until allblocks are deleted.

   * @return This RDD.

   */

  def unpersist(blocking: Boolean = true):this.type = {

    logInfo("Removing RDD " + id +" from persistence list")

    sc.unpersistRDD(id, blocking)

    storageLevel = StorageLevel.NONE

    this

  }

从上述的源码当中，我们可以清晰的看出，调用cache方法其实就是调用缓存策略StorageLevel值为MEMORY_ONLY类型的persist方法，所以才说cache是persist的特例。

我们这里顺便也说一下缓存级别：

先看一下源代码位置：

\spark-1.5.0\core\src\main\scala\org\apache\spark\storage\StorageLevel.scala

/**

 * Various[[org.apache.spark.storage.StorageLevel]] defined and utility functions forcreating

 * new storage levels.

 */

objectStorageLevel {

  val NONE = new StorageLevel(false, false,false, false)

  val DISK_ONLY = new StorageLevel(true, false,false, false)

  val DISK_ONLY_2 = new StorageLevel(true,false, false, false, 2)

  val MEMORY_ONLY = new StorageLevel(false,true, false, true)

  val MEMORY_ONLY_2 = new StorageLevel(false,true, false, true, 2)

  val MEMORY_ONLY_SER = new StorageLevel(false,true, false, false)

  val MEMORY_ONLY_SER_2 = newStorageLevel(false, true, false, false, 2)

  val MEMORY_AND_DISK = new StorageLevel(true,true, false, true)

  val MEMORY_AND_DISK_2 = newStorageLevel(true, true, false, true, 2)

  val MEMORY_AND_DISK_SER = newStorageLevel(true, true, false, false)

  val MEMORY_AND_DISK_SER_2 = newStorageLevel(true, true, false, false, 2)

  val OFF_HEAP = new StorageLevel(false, false,true, false) //专门针对Tachyon

……………………………………………..

 

// 这个类需要五个参数，可以看上面的伴生对象传入的参数

class StorageLevelprivate(

    private var _useDisk: Boolean,

    private var _useMemory: Boolean,

    privatevar _useOffHeap: Boolean, //是否使用tachyon分布式内存系统保存RDD

    private var _deserialized: Boolean, //kryo序列化比较好,节省内存，同时消耗cpu

    private var _replication: Int = 1)

  extends Externalizable {

 

  // TODO: Also add fields for cachingpriority, dataset ID, and flushing.

  private def this(flags: Int, replication:Int) {

    this((flags & 8) != 0, (flags & 4)!= 0, (flags & 2) != 0, (flags & 1) != 0, replication)

  }

 

  def this() = this(false, true, false,false)  // For deserialization

 

  def useDisk: Boolean = _useDisk

  def useMemory: Boolean = _useMemory

  def useOffHeap: Boolean = _useOffHeap

  def deserialized: Boolean = _deserialized

  def replication: Int = _replication

...............................................................................................

所以，我们也可以通过persist方法手工设定StorageLevel参数值来满足自己需要的缓存级别。