Job Submission failed with exception 'java.io.IOException

来源：互联网发布：c语言中变量命名规则编辑：程序博客网时间：2024/06/06 20:15

在hive上执行如下sql：

select a.product_id, a.merchant_id, b.category_id from pm_info a join product b on (a.product_id=b.id) where a.is_deleted=0 and a.product_is_gift=0 and a.can_sale=1 and a.can_show=1 and b.is_deleted=0 and b.PRODUCT_IS_GIFT=0;

出现错误，错误如下：

java.io.IOException: Failed to run job : org.apache.hadoop.security.AccessControlException: User pms cannot submit applications to queue root.defaultat org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:300)at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:396)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)at org.apache.hadoop.mapreduce.Job.submit(Job.java:1292)at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:396)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:448)at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138)at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1352)at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1138)at org.apache.hadoop.hive.ql.Driver.run(Driver.java:951)at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)at java.lang.reflect.Method.invoke(Method.java:597)at org.apache.hadoop.util.RunJar.main(RunJar.java:212)Job Submission failed with exception 'java.io.IOException(Failed to run job : org.apache.hadoop.security.AccessControlException: User pms cannot submit applications to queue root.default)'FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask

原因分析：执行的SQL语句含有where条件，join连接，就会执行MapReduce任务去执行，1号店新集群的MapReduce含有两个队列，一个是root.default队列，一个是pms队列；如果不指定提交到哪个队列，hadoop就会将此Job提交到默认的root.default队列中，但是集群设定了job由MapReduce的pms队列受理，所以提交到root.default队列的任务通通被打回

解决方案（指定Job提交到哪个MapReduce的队列，在shell中执行）：

hive -e "set mapred.job.queue.name=pms;select a.product_id, a.merchant_id, b.category_id from pm_info a join product b on (a.product_id=b.id) where a.is_deleted=0 and a.product_is_gift=0 and a.can_sale=1 and a.can_show=1 and b.is_deleted=0 and b.PRODUCT_IS_GIFT=0;"

5 0