spark 2.1 BlockManagerId
来源:互联网 发布:美团刷销量软件 编辑:程序博客网 时间:2024/06/16 08:38
When block manager is created, it will generate a id without topology information.
and register itself with the block manager master.
the block manager master will return a id, and it will use this id if the id is not null.
val id = BlockManagerId(executorId, blockTransferService.hostName, blockTransferService.port, None) val idFromMaster = master.registerBlockManager( id, maxMemory, slaveEndpoint) blockManagerId = if (idFromMaster != null) idFromMaster else id
BlockManagerMasterEndpoint.register method
private def register( idWithoutTopologyInfo: BlockManagerId, maxMemSize: Long, slaveEndpoint: RpcEndpointRef): BlockManagerId = { // the dummy id is not expected to contain the topology information. // we get that info here and respond back with a more fleshed out block manager id val id = BlockManagerId( idWithoutTopologyInfo.executorId, idWithoutTopologyInfo.host, idWithoutTopologyInfo.port, topologyMapper.getTopologyForHost(idWithoutTopologyInfo.host)) val time = System.currentTimeMillis() if (!blockManagerInfo.contains(id)) { blockManagerIdByExecutor.get(id.executorId) match { case Some(oldId) => // A block manager of the same executor already exists, so remove it (assumed dead) logError("Got two different block manager registrations on same executor - " + s" will replace old one $oldId with new one $id") removeExecutor(id.executorId) case None => } logInfo("Registering block manager %s with %s RAM, %s".format( id.hostPort, Utils.bytesToString(maxMemSize), id)) blockManagerIdByExecutor(id.executorId) = id blockManagerInfo(id) = new BlockManagerInfo( id, System.currentTimeMillis(), maxMemSize, slaveEndpoint) } listenerBus.post(SparkListenerBlockManagerAdded(time, id, maxMemSize)) id }
0 0
- spark 2.1 BlockManagerId
- spark 2.1 spark.yarn.services
- Spark 2.1.0 -- Spark Streaming Programming Guide
- spark 2.1 spark-shell Startup Process
- spark 2.1 spark executor topology information
- spark-2.1.0安装
- spark(2.1.0)
- Spark 2.1 structured streaming
- Spark 2.1 CallSite
- spark 2.1 ConfigProvider
- Spark ML 2.1 --Pipelines
- spark-2.1.0
- spark 2.1 SparkContext postEnvironmentUpdate
- spark 2.1 StorageLevel
- spark 2.1 BlockManagerMasterEndpoint
- spark 2.1 BlockManagerInfo
- spark 2.1 BlockManagerSlaveEndpoint
- spark 2.1 Memory Store
- java使用Arrays asList需要注意了
- MySQL EXISTS /NOT EXISTS
- 数据库特征值的替换
- YII post提交报错
- 失恋了,开始写博客吧
- spark 2.1 BlockManagerId
- 聚合数据 调用API(违章查询)
- 解决springMVC无法接受日期类型参数
- 我的mac下iterm实用快捷键
- C++ 头文件保护符
- c++中指针与引用的区别
- 图管够!灌篮高手、女儿国…阿里日_这帮程序员太会玩了!
- svn命令在linux下的使用
- python读写文件知识点