nodemanager报错 spark.network.yarn.YarnShuffleService not found

来源：互联网发布：中世纪2全面战争原版优化9 编辑：程序博客网时间：2024/05/16 19:04

HDP2.4版本

安装启动后nodemanager报错，无法启动，查看日志发现，

2016-03-11 10:51:23,638 FATAL nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(540)) - Error starting NodeManager
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:121)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:245)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:291)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:537)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:585)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2219)
        ... 10 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.spark.network.yarn.YarnShuffleService not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
        ... 11 more
2016-03-11 10:51:23,640 INFO nodemanager.NodeManager (LogAdapter.java:info(45)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at mapp3
************************************************************/

分析解决：
网上搜索了下，缺少org.apache.spark.network.yarn.YarnShuffleService相关的jar包文件，没有 spark-yarn-shuffle.jar包
查看正常节点上的spark*.jar包，

[root@map1 ~]# find / -name spark*.jar
/usr/hdp/2.4.0.0-169/spark/lib/spark-1.6.0.2.4.0.0-169-yarn-shuffle.jar
/usr/hdp/2.4.0.0-169/hadoop/lib/spark-yarn-shuffle.jar

异常节点上没有这个包
[root@map3 ~]# find / -name spark*.jar
[root@map3 ~]#

将jar拷贝到异常节点
[root@map1 ~]#scp /usr/hdp/2.4.0.0-169/hadoop/lib/spark-yarn-shuffle.jar 172.16.16.41:/usr/hdp/2.4.0.0-169/hadoop/lib/

启动nodemanager正常了，这个包应该是spark on yarn上需要使用。

0 0