Initial job has not accepted any resources; check your cluster UI to ensure that workers are registe

来源:互联网 发布:python ui界面开发 编辑:程序博客网 时间:2024/06/06 03:33

spark在提交任务时,出现如下错误:

15/03/26 22:29:36 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory15/03/26 22:29:51 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory15/03/26 22:30:06 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory15/03/26 22:30:21 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory


从警告信息上看,初始化job时没有获取到任何资源;提示检查集群,确保workers可以被注册并有足够的内存资源。


如上问题产生的原因是多方面的,可能原因如下:


1.因为提交任务的节点不能和spark工作节点交互,因为提交完任务后提交任务节点上会起一个进程,展示任务进度,大多端口为4044,工作节点需要反馈进度给该该端口,所以如果主机名或者IP在hosts中配置不正确。所以检查下主机名和ip是否配置正确


2.有可能是内存不足

  检查内存

   conf.set("spark.executor.memory", "3000m")

  Make sure to set SPARK_LOCAL_IP andSPARK_MASTER_IP.

  查看8080端口,确保一些workers保持Alive状态,确保 some cores 是可利用的。

0 0