hadoop（2.7.1）配置过程问题

来源：互联网发布：苏州楼市成交数据编辑：程序博客网时间：2024/05/17 00:11

1. formate 出错

ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Cannot create directory /usr/hadoop/dfs/name/current
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDi rectory(Storage.java:337)

解决方法：/usr/hadoop/dfs 和 /usr/hadoop/tmp 都没有权限，更改如下

sudo chown -R hadoop:hadoop /usr/hadoop/dfs
sudo chown -R hadoop:hadoop /usr/hadoop/tmp

2. start-all.sh 启动登陆密码

设置完免密码后，master 登陆master 仍然需要登陆密码，
这里写图片描述
解决方法为：
将id_rsa.pub 公钥内容，拷贝到 authorized_keys 中。

3. hdfs上传文件出错

17/03/02 21:56:02 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /data/input/REA DME.txt.COPYING could only be replicated to 0 nodes instead of minReplication (=1). There are 0 datanode(s) running and no node(s) are excluded in this opera tion.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarg et4NewBlock(BlockManager.java:1550)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTarget s(FSNamesystem.java:3110)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBloc k(FSNamesystem.java:3034)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(Nam eNodeRpcServer.java:723)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTra nslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtosClientNamenodeProtocol2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngineServerProtoBufRpcInvoker.cal l(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPCServer.call(RPC.java:969)atorg.apache.hadoop.ipc.ServerHandler1.run(Server.java:2049)atorg.apache.hadoop.ipc.ServerHandler1.run(Server.java:2045)atjava.security.AccessController.doPrivileged(NativeMethod)atjavax.security.auth.Subject.doAs(Subject.java:422)atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)atorg.apache.hadoop.ipc.ServerHandler.run(Server.java:2043)

    at org.apache.hadoop.ipc.Client.call(Client.java:1476)    at org.apache.hadoop.ipc.Client.call(Client.java:1407)    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEng                                                                                                             ine.java:229)    at com.sun.proxy.$Proxy9.addBlock(Unknown Source)    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.                                                                                                             addBlock(ClientNamenodeProtocolTranslatorPB.java:418)    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.                                                                                                             java:62)    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces                                                                                                             sorImpl.java:43)    at java.lang.reflect.Method.invoke(Method.java:498)    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryI                                                                                                             nvocationHandler.java:187)    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocat                                                                                                             ionHandler.java:102)    at com.sun.proxy.$Proxy10.addBlock(Unknown Source)    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBl                                                                                                             ock(DFSOutputStream.java:1430)    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputSt                                                                                                             ream(DFSOutputStream.java:1226)    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStre                                                                                                             am.java:449)

put: File /data/input/README.txt.COPYING could only be replicated to 0 nodes i nstead of minReplication (=1). There are 0 datanode(s) running and no node(s) a re excluded in this operation.

方法：

1、关闭防火墙： sudo ufw disable
2、将/etc/hosts 文件下， #127.0.1.1 master 屏蔽掉
3、start-dfs.sh start-yarn.sh

0 0