spark-broadcast

@(spark)[broadcast]
Spark’s broadcast variables, used to broadcast immutable datasets to all node

Broadcast

/**                                                                                                                                                                      * A broadcast variable. Broadcast variables allow the programmer to keep a read-only variable                                                                           * cached on each machine rather than shipping a copy of it with tasks. They can be used, for                                                                            * example, to give every node a copy of a large input dataset in an efficient manner. Spark also                                                                        * attempts to distribute broadcast variables using efficient broadcast algorithms to reduce                                                                             * communication cost.                                                                                                                                                   *                                                                                                                                                                       * Broadcast variables are created from a variable `v` by calling                                                                                                        * [[org.apache.spark.SparkContext#broadcast]].                                                                                                                          * The broadcast variable is a wrapper around `v`, and its value can be accessed by calling the                                                                          * `value` method. The interpreter session below shows this:                                                                                                             *                                                                                                                                                                       * {{{                                                                                                                                                                   * scala> val broadcastVar = sc.broadcast(Array(1, 2, 3))                                                                                                                * broadcastVar: org.apache.spark.broadcast.Broadcast[Array[Int]] = Broadcast(0)                                                                                         *                                                                                                                                                                       * scala> broadcastVar.value                                                                                                                                             * res0: Array[Int] = Array(1, 2, 3)                                                                                                                                     * }}}                                                                                                                                                                   *                                                                                                                                                                       * After the broadcast variable is created, it should be used instead of the value `v` in any                                                                            * functions run on the cluster so that `v` is not shipped to the nodes more than once.                                                                                  * In addition, the object `v` should not be modified after it is broadcast in order to ensure                                                                           * that all nodes get the same value of the broadcast variable (e.g. if the variable is shipped                                                                          * to a new node later).                                                                                                                                                 *                                                                                                                                                                       * @param id A unique identifier for the broadcast variable.                                                                                                             * @tparam T Type of the data contained in the broadcast variable.                                                                                                       */                                                                                                                                                                     abstract class Broadcast[T: ClassTag](val id: Long) extends Serializable with Logging {    /**                                                                                                                                                                      * :: DeveloperApi ::                                                                                                                                                    * An interface for all the broadcast implementations in Spark (to allow                                                                                                 * multiple broadcast implementations). SparkContext uses a user-specified                                                                                               * BroadcastFactory implementation to instantiate a particular broadcast for the                                                                                         * entire Spark job.                                                                                                                                                     */                                                                                                                                                                     @DeveloperApi                                                                                                                                                           trait BroadcastFactory {   

目前有两组实现，默认的是后者

HttpBroadcast

/**                                                                                                                                                                      * A [[org.apache.spark.broadcast.Broadcast]] implementation that uses HTTP server                                                                                       * as a broadcast mechanism. The first time a HTTP broadcast variable (sent as part of a                                                                                 * task) is deserialized in the executor, the broadcasted data is fetched from the driver                                                                                * (through a HTTP server running at the driver) and stored in the BlockManager of the                                                                                   * executor to speed up future accesses.                                                                                                                                 */                                                                                                                                                                     private[spark] class HttpBroadcast[T: ClassTag](     

TorrentBroadcast

/**                                                                                                                                                                      * A BitTorrent-like implementation of [[org.apache.spark.broadcast.Broadcast]].                                                                                         *                                                                                                                                                                       * The mechanism is as follows:                                                                                                                                          *                                                                                                                                                                       * The driver divides the serialized object into small chunks and                                                                                                        * stores those chunks in the BlockManager of the driver.                                                                                                                *                                                                                                                                                                       * On each executor, the executor first attempts to fetch the object from its BlockManager. If                                                                           * it does not exist, it then uses remote fetches to fetch the small chunks from the driver and/or                                                                       * other executors if available. Once it gets the chunks, it puts the chunks in its own                                                                                  * BlockManager, ready for other executors to fetch from.                                                                                                                *                                                                                                                                                                       * This prevents the driver from being the bottleneck in sending out multiple copies of the                                                                              * broadcast data (one per executor) as done by the [[org.apache.spark.broadcast.HttpBroadcast]].                                                                        *                                                                                                                                                                       * When initialized, TorrentBroadcast objects read SparkEnv.get.conf.                                                                                                    *                                                                                                                                                                       * @param obj object to broadcast                                                                                                                                        * @param id A unique identifier for the broadcast variable.                                                                                                             */                                                                                                                                                                     private[spark] class TorrentBroadcast[T: ClassTag](obj: T, id: Long)                                                                                                      extends Broadcast[T](id) with Logging with Serializable {

随机选远程节点这个事情，是由blockManger完成的