Spark RpcEnv

在SparkContext初始化环境时使用下面代码创建RpcEnv

val rpcEnv = RpcEnv.create(actorSystemName, hostname, port, conf, securityManager,

  clientMode = !isDriver)

创建RpcEnv时通过创建RpcEnvFactory(默认为netty)然后使用工厂创建RpcEnv,如下代码所示

val config = RpcEnvConfig(conf, name, host, port, securityManager, clientMode)

getRpcEnvFactory(conf).create(config)

创建RpcEnvFactory代码如下：

vate def getRpcEnvFactory(conf: SparkConf): RpcEnvFactory = {

val rpcEnvNames = Map(

"akka" -> "org.apache.spark.rpc.akka.AkkaRpcEnvFactory",

"netty" -> "org.apache.spark.rpc.netty.NettyRpcEnvFactory")

val rpcEnvName = conf.get("spark.rpc", "netty")

val rpcEnvFactoryClassName = rpcEnvNames.getOrElse(rpcEnvName.toLowerCase, rpcEnvName)      Utils.classForName(rpcEnvFactoryClassName).newInstance().asInstanceOf[RpcEnvFactory]

}

下面为创建RpcEnv的细节，以NettyRpcEnv为例

a.初始化JavaSerializer，初始化NettyRpcEnv，如果是 非客户端模式就启动netty服务 

val javaSerializerInstance =

  new JavaSerializer(sparkConf).newInstance().asInstanceOf[JavaSerializerInstance]

val nettyEnv =

  new NettyRpcEnv(sparkConf, javaSerializerInstance, config.host, config.securityManager)

b.启动NettyServer

创建List<SaslServerBootStrap>，SaslServerBootStrap继承自TransportServerBootstrap,同时初始化SparkTransportConf，

java.util.Arrays.asList(new SaslServerBootstrap(transportConf, securityManager))

SparkTransportConf拷贝了SparkConfig同时 new TransportConf,如下代码所示

def fromSparkConf(_conf: SparkConf, module: String, numUsableCores: Int = 0): TransportConf = {

  val conf = _conf.clone

.....

  new TransportConf(module, new ConfigProvider {

    override def get(name: String): String = conf.get(name)

  })

}

实例化TransportConf的代码如下：

public TransportConf(String module, ConfigProvider conf) {

    //rpc

this.module = module;

this.conf = conf;

this.SPARK_NETWORK_IO_MODE_KEY = this.getConfKey("io.mode");

this.SPARK_NETWORK_IO_PREFERDIRECTBUFS_KEY = this.getConfKey("io.preferDirectBufs");

this.SPARK_NETWORK_IO_CONNECTIONTIMEOUT_KEY = this.getConfKey("io.connectionTimeout");

this.SPARK_NETWORK_IO_BACKLOG_KEY = this.getConfKey("io.backLog");

this.SPARK_NETWORK_IO_NUMCONNECTIONSPERPEER_KEY = this.getConfKey("io.numConnectionsPerPeer");

this.SPARK_NETWORK_IO_SERVERTHREADS_KEY = this.getConfKey("io.serverThreads"); 

this.SPARK_NETWORK_IO_CLIENTTHREADS_KEY = this.getConfKey("io.clientThreads");

this.SPARK_NETWORK_IO_RECEIVEBUFFER_KEY = this.getConfKey("io.receiveBuffer");

this.SPARK_NETWORK_IO_SENDBUFFER_KEY = this.getConfKey("io.sendBuffer");

this.SPARK_NETWORK_SASL_TIMEOUT_KEY = this.getConfKey("sasl.timeout");

this.SPARK_NETWORK_IO_MAXRETRIES_KEY = this.getConfKey("io.maxRetries");

this.SPARK_NETWORK_IO_RETRYWAIT_KEY = this.getConfKey("io.retryWait");

this.SPARK_NETWORK_IO_LAZYFD_KEY = this.getConfKey("io.lazyFD");

}

private String getConfKey(String suffix) {

return "spark." + this.module + "." + suffix;

}

使用transportContext创建TransportServer

server = transportContext.createServer(host, port, bootstraps)

public TransportServer createServer(String host, int port, List<TransportServerBootstrap> bootstraps) {
return new TransportServer(this, host, port, this.rpcHandler, bootstraps);
}

public TransportServer(TransportContext context, String hostToBind, int portToBind, RpcHandler appRpcHandler, List<TransportServerBootstrap> bootstraps) {
...
this.init(hostToBind, portToBind);
    ...
}

init方法如下

通过配置<spark.rpc.io.mode>获取ioMode实例,默认为nio

IOMode ioMode = IOMode.valueOf(this.conf.ioMode());

EventLoopGroup bossGroup = NettyUtils.createEventLoop(ioMode, this.conf.serverThreads(), "shuffle-server");

通过配置<spark.rpc.io.serverThreads>初始化serverThreads的个数，默认为0,然后创建EventLoop,默认eventLoop使用线程池的大小

Math.max(1, SystemPropertyUtil.getInt("io.netty.eventLoopThreads", Runtime.getRuntime().availableProcessors() * 2))

PooledByteBufAllocator allocator = NettyUtils.createPooledByteBufAllocator(this.conf.preferDirectBufs(), true, 

this.conf.serverThreads());

通过配置<spark.rpc.io.preferDirectBufs>初始化preferDirectBufs，默认为true

新建
bootstrap并初始化
bootstrap = new ServerBootstrap()
  .group(bossGroup, workerGroup)
  .channel(NettyUtils.getServerChannelClass(ioMode))
  .option(ChannelOption.ALLOCATOR, allocator)
  .childOption(ChannelOption.ALLOCATOR, allocator);

....为bootstrap设置Options...

bootstrap.childHandler(new ChannelInitializer<SocketChannel>() {
@Override
protected void initChannel(SocketChannel ch) throws Exception {
    RpcHandler rpcHandler = appRpcHandler;
//遍历TransportServer中的每个
bootstrap并生成
SaslRpcHandler
for (TransportServerBootstrap bootstrap : bootstraps) {
      rpcHandler = bootstrap.doBootstrap(ch, rpcHandler);
    }

context.initializePipeline(ch, rpcHandler);
  }
});

InetSocketAddress address = hostToBind == null ?
new InetSocketAddress(portToBind): new InetSocketAddress(hostToBind, portToBind);
//绑定地址
channelFuture = bootstrap.bind(address);
//使得绑定地址的过程变得同步不可中断
channelFuture.syncUninterruptibly();

port = ((InetSocketAddress) channelFuture.channel().localAddress()).getPort();
===============================================================================================

context.initializePipeline(ch, rpcHandler)代码如下

public TransportChannelHandler initializePipeline(
    SocketChannel channel,
    RpcHandler channelRpcHandler) {
//生成ChannelHandler
    TransportChannelHandler channelHandler = createChannelHandler(channel, channelRpcHandler);
    //为channel添加参数
    channel.pipeline()
      .addLast("encoder", encoder)
      .addLast(TransportFrameDecoder.HANDLER_NAME, NettyUtils.createFrameDecoder())
      .addLast("decoder", decoder)
      .addLast("idleStateHandler", new IdleStateHandler(0, 0, conf.connectionTimeoutMs() / 1000))
.addLast("handler", channelHandler);
return channelHandler;

}
private TransportChannelHandler createChannelHandler(Channel channel, RpcHandler rpcHandler) {
 //根据channel创建ResponseHandler
  TransportResponseHandler responseHandler = new TransportResponseHandler(channel);
//根据channel和responseHandler创建client
  TransportClient client = new TransportClient(channel, responseHandler);
//根据channel,client,rpcHandler创建requestHandler
  TransportRequestHandler requestHandler = new TransportRequestHandler(channel, client,
    rpcHandler);
return new TransportChannelHandler(client, responseHandler, requestHandler,
conf.connectionTimeoutMs(), closeIdleConnections);
->向分发器注册该RpcEndpoint
dispatcher.registerRpcEndpoint(
  RpcEndpointVerifier.NAME, new RpcEndpointVerifier(this, dispatcher))
Dispatcher
private val endpoints = new ConcurrentHashMap[String, EndpointData]
private val endpointRefs = new ConcurrentHashMap[RpcEndpoint, RpcEndpointRef]

  RpcEndpointVerifier.NAME, new RpcEndpointVerifier(this, dispatcher))