问题描述：三台机子搭建的hadoop集群，一台是namenode，另外两台是datanode。今天执行hadoop fs -copyFromLocal 的时候报错。File /home/hexianghui/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 由这个可以认为是没有找到datanode。我用jps查看进程都正常。但是用web查看的话，live nodes为0. 这说明datanode没有正常启动，但是datanode进程又启动了，这是为何?网友可以跟帖提出意见或者指导……
2009-12-30 22:02:19,190 INFO org.apache.hadoop.mapred.JobTracker: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = hexianghui/192.168.0.4
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.1
STARTUP_MSG: build =http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.20.1-rc1 -r 810220; compiled by 'oom' on Tue Sep 1 20:55:56 UTC 2009
************************************************************/
2009-12-30 22:02:19,280 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2009-12-30 22:02:19,322 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=JobTracker, port=9001
2009-12-30 22:02:24,544 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2009-12-30 22:02:24,691 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50030
2009-12-30 22:02:24,692 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50030 webServer.getConnectors()[0].getLocalPort() returned 50030
2009-12-30 22:02:24,692 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50030
2009-12-30 22:02:24,692 INFO org.mortbay.log: jetty-6.1.14
2009-12-30 22:04:01,351 INFO org.mortbay.log: StartedSelectChannelConnector@0.0.0.0:50030
2009-12-30 22:04:01,353 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
2009-12-30 22:04:01,353 INFO org.apache.hadoop.mapred.JobTracker: JobTracker up at: 9001
2009-12-30 22:04:01,353 INFO org.apache.hadoop.mapred.JobTracker: JobTracker webserver: 50030
2009-12-30 22:04:01,468 INFO org.apache.hadoop.mapred.JobTracker: Cleaning up the system directory
2009-12-30 22:04:01,511 INFO org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is inactive
2009-12-30 22:04:01,555 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /home/hexianghui/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy4.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
2009-12-30 22:04:01,555 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null
2009-12-30 22:04:01,555 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Source file "/home/hexianghui/tmp/mapred/system/jobtracker.info" - Aborting...
2009-12-30 22:04:01,556 WARN org.apache.hadoop.mapred.JobTracker: Writing to file hdfs://hexianghui:9000/home/hexianghui/tmp/mapred/system/jobtracker.info failed!
2009-12-30 22:04:01,556 WARN org.apache.hadoop.mapred.JobTracker: FileSystem is not ready yet!
2009-12-30 22:04:01,568 WARN org.apache.hadoop.mapred.JobTracker: Failed to initialize recovery manager.
org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /home/hexianghui/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
at org.apache.hadoop.ipc.Client.call(Client.java:739)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy4.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy4.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2904)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2786)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
2009-12-30 22:04:11,569 WARN org.apache.hadoop.mapred.JobTracker: Retrying...
以下是我在网上找的解决方法集锦：
方案1：
是否是防火墙未关闭，查看。确实忘记关闭防火墙了，因为我换了几台机器，以前是在虚拟机下用的是redhat，现在用的是ubuntu8.0.4。所以跟这个很可能相关。关闭iptables后，出现的错误信息没那么多了，但是还有错误。如下：
2010-01-03 22:08:25,073 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 9000, call addBlock(/user/hexianghui/input/file01, DFSClient_105866075) from 192.168.0.4:53604: error: java.io.IOException: File /user/hexianghui/input/file01 could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /user/hexianghui/input/file01 could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1267)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:422)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
2010-01-03 22:08:25,087 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=hexianghui,hexianghui,adm,dialout,cdrom,floppy,audio,dip,video,plugdev,fuse,lpadmin,admin ip=/192.168.0.4 cmd=create src=/user/hexianghui/input/file02 dst=null perm=hexianghui:supergroup:rw-r--r--
2010-01-03 22:08:25,088 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9000, call addBlock(/user/hexianghui/input/file02, DFSClient_105866075) from 192.168.0.4:53604: error: java.io.IOException: File /user/hexianghui/input/file02 could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /user/hexianghui/input/file02 could only be replicated to 0 nodes, instead of 1
虽然有类似的信息：could only be replicated to 0 nodes, instead of 1。但情况不一样了，可以对照上面的错误信息。继续接着找……
方案2：Hadoop DFSClient警告NotReplicatedYetException信息
有时，当你申请到一个HOD集群后马上尝试上传文件到HDFS时，DFSClient会警告NotReplicatedYetException。通常会有一个这样的信息 -
WARN hdfs.DFSClient: NotReplicatedYetException sleeping <filename> retries left 308/01/25 16:31:40 INFO hdfs.DFSClient: org.apache.hadoop.ipc.RemoteException:java.io.IOException: File <filename>could only be replicated to 0 nodes, instead of 1当你向一个DataNodes正在和NameNode联络的集群上传文件的时候，这种现象就会发生。在上传新文件到HDFS之前多等待一段时间就可以解决这个问题，因为这使得足够多的DataNode启动并且联络上了NameNode。
PS:我的实践：等待了几分钟，还是依然报错，此法不通。
方案3：http://trac.nchc.org.tw/cloud/wiki/waue/2009/0709
這個錯誤訊息意思是，他想要放檔案，但沒半個node可以給存取，因此我們需要檢查：
- 系統或hdfs是否還有空間 （像我就是）
- datanode數是否正常
- 是否在safemode
- 讀寫的權限
PS:检查上面几个，1）系统空间够。df -hl查看。 2）datanode数是2.datanode用jps查看进程，都启动了。3，是否在safemode下。hadoop dfsadmin -safemode leave.使用后，可以正常拷贝了。也许之前的操作和这一步操作起作用了。
转自：http://blog.163.com/zhengjiu_520/blog/static/3559830620118573855465/
