Hama设置hadooprep=3时出现问题

hadoop datanode日志:

2016-02-25 18:11:23,000 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving blk_6574808912059483624_1989 src: /192.168.0.29:54298 dest: /192.168.0.25:50010
2016-02-25 18:11:23,001 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.0.25:50010, storageID=DS-1710700403-192.168.0.25-50010-1456387922287, infoPort=50075, ipcPort=50020):DataXceiver
java.lang.NegativeArraySizeException
at org.apache.hadoop.io.Text.readString(Text.java:401)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:253)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:112)
at java.lang.Thread.run(Thread.java:745)

hama任务日志:

attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 INFO hdfs.DFSClient: Abandoning blk_-3990355810599177962_2089
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 INFO hdfs.DFSClient: Excluding datanode 192.168.0.24:50010
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 WARN hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Unable to create new block.
attempt_201602251809_0006_000008_1:     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3615)
attempt_201602251809_0006_000008_1:     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$9(DFSClient.java:3572)
attempt_201602251809_0006_000008_1:     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3027)
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 WARN hdfs.DFSClient: Error Recovery for blk_-3990355810599177962_2089 bad datanode[0] nodes == null
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 WARN hdfs.DFSClient: Could not get block locations. Source file "/tmp/hama-parts/job_201602251809_0005/part-6/file-8" - Aborting...
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 ERROR bsp.BSPTask: Error running bsp setup and bsp function.
attempt_201602251809_0006_000008_1: java.io.EOFException
attempt_201602251809_0006_000008_1:     at java.io.DataInputStream.readShort(DataInputStream.java:315)
attempt_201602251809_0006_000008_1:     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:3684)
attempt_201602251809_0006_000008_1:     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3598)
attempt_201602251809_0006_000008_1:     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$9(DFSClient.java:3572)
attempt_201602251809_0006_000008_1:     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3027)
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 INFO ipc.Server: Stopping server on 61001
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 INFO ipc.Server: IPC Server handler 0 on 61001: exiting
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 INFO ipc.Server: IPC Server handler 1 on 61001: exiting
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 INFO ipc.Server: IPC Server handler 2 on 61001: exiting
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 INFO ipc.Server: IPC Server handler 4 on 61001: exiting
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 DEBUG ipc.Server: Checking for old call responses.
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 INFO ipc.Server: Stopping IPC Server listener on 61001
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 ERROR bsp.BSPTask: Shutting down ping service.
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 INFO ipc.Server: Stopping IPC Server Responder
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 INFO ipc.Server: IPC Server handler 3 on 61001: exiting
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 FATAL bsp.GroomServer: Error running child
attempt_201602251809_0006_000008_1: java.io.EOFException
attempt_201602251809_0006_000008_1:     at java.io.DataInputStream.readShort(DataInputStream.java:315)
attempt_201602251809_0006_000008_1:     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:3684)
attempt_201602251809_0006_000008_1:     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3598)
attempt_201602251809_0006_000008_1:     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$9(DFSClient.java:3572)
attempt_201602251809_0006_000008_1:     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3027)
attempt_201602251809_0006_000008_1: java.io.EOFException
attempt_201602251809_0006_000008_1:     at java.io.DataInputStream.readShort(DataInputStream.java:315)
attempt_201602251809_0006_000008_1:     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:3684)
attempt_201602251809_0006_000008_1:     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3598)
attempt_201602251809_0006_000008_1:     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$9(DFSClient.java:3572)
attempt_201602251809_0006_000008_1:     at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3027)
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 DEBUG ipc.Client: Stopping client
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:50001 from liuqiang2: closed
attempt_201602251809_0006_000008_1: 16/02/25 18:10:14 DEBUG ipc.Client: IPC Client (47) connection to localhost/127.0.0.1:50001 from liuqiang2: stopped, remaining connections 0

初步怀疑是DFSCLient的字符串问题，可能是在写入块时哪里出错了!

 对比下Client ,怀疑是哪个标志位错了！ 

奇怪的是，在增加打印日志后，同一个作业在不同节点上的一个任务日志能够顺利打印，另一个不行:

c02

16/02/25 20:16:43 DEBUG bsp.PartitioningRunner:  FInish getPartitioner()
16/02/25 20:16:44 INFO hdfs.DFSClient:  nodes.length 3
16/02/25 20:16:44 INFO hdfs.DFSClient:  recoveryFlag false
16/02/25 20:16:44 INFO hdfs.DFSClient:  client DFSClient_NONMAPREDUCE_2019487700_1

c04

16/02/25 20:13:25 INFO hdfs.DFSClient: Exception in createBlockOutputStream 192.168.0.24:50010 java.io.EOFException
16/02/25 20:13:25 INFO hdfs.DFSClient: Abandoning blk_-4196437268512905844_3326
16/02/25 20:13:25 INFO hdfs.DFSClient: Excluding datanode 192.168.0.24:50010
16/02/25 20:13:25 INFO hdfs.DFSClient: Exception in createBlockOutputStream 192.168.0.27:50010 java.io.EOFException
16/02/25 20:13:25 INFO hdfs.DFSClient: Abandoning blk_-2182714137940309922_3326
16/02/25 20:13:25 INFO hdfs.DFSClient: Excluding datanode 192.168.0.27:50010
16/02/25 20:13:25 INFO hdfs.DFSClient: Exception in createBlockOutputStream 192.168.0.22:50010 java.io.EOFException
16/02/25 20:13:25 INFO hdfs.DFSClient: Abandoning blk_1951775927120661581_3326
16/02/25 20:13:25 INFO hdfs.DFSClient: Excluding datanode 192.168.0.22:50010
16/02/25 20:13:25 INFO hdfs.DFSClient: Exception in createBlockOutputStream 192.168.0.30:50010 java.io.EOFException
16/02/25 20:13:25 INFO hdfs.DFSClient: Abandoning blk_-2720588266343979701_3326
16/02/25 20:13:25 INFO hdfs.DFSClient: Excluding datanode 192.168.0.30:50010
16/02/25 20:13:25 WARN hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Unable to create new block.
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3615)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$9(DFSClient.java:3572)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:3027)


16/02/25 20:13:25 WARN hdfs.DFSClient: Error Recovery for blk_-2720588266343979701_3326 bad datanode[0] nodes == null
16/02/25 20:13:25 WARN hdfs.DFSClient: Could not get block locations. Source file "/tmp/hama-parts/job_201602252016_0005/part-8/file-5" - Aborting...
16/02/25 20:13:25 ERROR bsp.BSPTask: Error running bsp setup and bsp function.
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:3684)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3598)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$9(DFSClient.java:3572)

似乎该任务刚启动DFSOutXXStream即报错！ 好诡异啊