Error: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-965200530-172.21.3.1

跑hive脚本也就是mr程序时遇到以下错误：

Ended Job = job_1406698610363_0394 with errors

Error during job, obtaining debugging information...

Examining task ID: task_1406698610363_0394_m_000014 (and more) from job job_1406698610363_0394

Examining task ID: task_1406698610363_0394_m_000006 (and more) from job job_1406698610363_0394

Examining task ID: task_1406698610363_0394_m_000022 (and more) from job job_1406698610363_0394

Examining task ID: task_1406698610363_0394_m_000030 (and more) from job job_1406698610363_0394

Examining task ID: task_1406698610363_0394_m_000034 (and more) from job job_1406698610363_0394

Examining task ID: task_1406698610363_0394_m_000040 (and more) from job job_1406698610363_0394

Examining task ID: task_1406698610363_0394_m_000041 (and more) from job job_1406698610363_0394

Examining task ID: task_1406698610363_0394_m_000039 (and more) from job job_1406698610363_0394

 

Task with the most failures(4): 

-----

Task ID:

  task_1406698610363_0394_m_000008

 

URL:

  http://e3basemaster2:50030/taskdetails.jsp?jobid=job_1406698610363_0394&tipid=task_1406698610363_0394_m_000008

-----

Diagnostic Messages for this Task:

Error: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-965200530-172.21.3.170-1400216975207:blk_-1762002543329523353_80392 file=/user/bdsdata/.staging/job_1406698610363_0394/job.split

        at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:734)

        at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:448)

        at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:645)

        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:689)

        at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:518)

        at java.io.DataInputStream.readByte(DataInputStream.java:248)

        at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)

        at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)

        at org.apache.hadoop.io.Text.readString(Text.java:446)

        at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:345)

        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:375)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)

        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:396)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)

        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)

 

 

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask

MapReduce Jobs Launched: 

Job 0: Map: 159  Reduce: 10   Cumulative CPU: 551.26 sec   HDFS Read: 2885868030 HDFS Write: 0 FAIL

Total MapReduce CPU Time Spent: 9 minutes 11 seconds 260 msec

错误很明显无法获得BP-965200530-172.21.3.170-1400216975207:blk_-1762002543329523353_80392 file=/user/bdsdata/.staging/job_1406698610363_0394/job.split

的数据，然后节点监控页面发现datanode已经死了两个（本来有三个datanode两个namenode），然后看了下hdfs-site.xml发现replication的值为2，可见是由于节点坏死导致无法获取数据，重启死掉的节点问题解决，当然也有可能是datanode的通信出现问题，但可能性比较小。