Spark1.5的一个bug

来源：互联网发布：手机扫描文字识别软件编辑：程序博客网时间：2024/06/05 02:14

>>> 16/10/15 20:07:35 INFO YarnClientSchedulerBackend: Requesting to kill executor(s) 116/10/15 20:07:35 INFO ExecutorAllocationManager: Removing executor 1 because it has been idle for 60 seconds (new desired total will be 0)16/10/15 20:07:36 ERROR YarnScheduler: Lost executor 1 on hadoop05: remote Rpc client disassociated16/10/15 20:07:36 INFO DAGScheduler: Executor lost: 1 (epoch 0)16/10/15 20:07:36 INFO BlockManagerMasterEndpoint: Trying to remove executor 1 from BlockManagerMaster.16/10/15 20:07:36 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(1, hadoop05, 41258)16/10/15 20:07:36 INFO BlockManagerMaster: Removed 1 successfully in removeExecutor16/10/15 20:07:36 INFO ExecutorAllocationManager: Existing executor 1 has been removed (new total is 0)

时不时就报ERROR YarnScheduler: Lost executor 1 on hadoop05: remote Rpc client disassociated的错误。

后查证该问题是spark1.5的bug由于启用了动态分配以及回收资源，当正确的回收资源后，会报出这个错误。
这个错误不会影响集群以及计算任务的结果。
Jira地址：https://issues.apache.org/jira/browse/SPARK-4134

最好的办法是将spark升级至1.6

0 0