Exception in createBlockOutputStream java.net.SocketException: 打开的文件过多

来源：互联网发布：萧朗作品知多少钱编辑：程序博客网时间：2024/06/17 05:28

执行MapReduce程序报如下错误：

15/03/29 22:35:13 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.net.SocketException: 打开的文件过多
   at sun.nio.ch.Net.socket0(Native Method)
   at sun.nio.ch.Net.socket(Net.java:423)
   at sun.nio.ch.Net.socket(Net.java:416)
   at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:104)
   at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
   at java.nio.channels.SocketChannel.open(SocketChannel.java:142)
   at org.apache.hadoop.net.StandardSocketFactory.createSocket(StandardSocketFactory.java:62)
   at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1608)
   at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408)
   at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1281)
   at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:594)
15/03/29 22:35:13 WARN hdfs.DFSClient: DataStreamer Exception
java.lang.NullPointerException
   at org.apache.hadoop.hdfs.DFSOutputStream$Packet.writeTo(DFSOutputStream.java:321)
   at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:643)
15/03/29 22:35:13 WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block BP-1306947336-127.0.1.1-1427186980802:blk_1073741941_5216
java.lang.NullPointerException
   at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2201)
   at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
   at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:867)
15/03/29 22:35:13 INFO mapred.MapTask: Starting flush of map output
15/03/29 22:35:13 INFO mapred.MapTask: Spilling map output
15/03/29 22:35:13 INFO mapred.MapTask: bufstart = 0; bufend = 16070994; bufvoid = 104857600
15/03/29 22:35:13 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 15500404(62001616); length = 10713993/6553600
15/03/29 22:35:14 INFO mapred.LocalJobRunner: map > map
15/03/29 22:35:17 INFO mapred.LocalJobRunner: map > map
15/03/29 22:35:19 INFO mapred.MapTask: Finished spill 0
15/03/29 22:35:19 INFO mapred.LocalJobRunner: map task executor complete.
15/03/29 22:35:19 WARN mapred.LocalJobRunner: job_local592353110_0001
java.lang.Exception: java.io.IOException: All datanodes 127.0.0.1:50010 are bad. Aborting...
   at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
   at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.io.IOException: All datanodes 127.0.0.1:50010 are bad. Aborting...
   at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1206)
   at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:594)
15/03/29 22:35:19 INFO mapreduce.Job: Job job_local592353110_0001 failed with state FAILED due to: NA
15/03/29 22:35:19 INFO mapreduce.Job: Counters: 25

原因：网络请求过多和打开的文件过多，没有释放一些文件句柄，程序打开了文件却没有在最后close。

0 0