hdfs报错之 DisallowedDatanodeException

先看异常堆栈,以前一直没有去看namenode的日志 今天已查看 发现一个错误 看错误的信息应该是无法解析到207机器

2017-10-10 09:45:45,124 WARN  blockmanagement.DatanodeManager (DatanodeManager.java:registerDatanode(882)) - Unresolved datanode registration: hostname cannot be resolved (ip=192.168.5.208, hostname=192.168.5.208)2017-10-10 09:45:45,124 INFO  ipc.Server (Server.java:logException(2361)) - IPC Server handler 106 on 8020, call org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.registerDatanode from 192.168.5.208:44010 Call#139663 Retry#0org.apache.hadoop.hdfs.server.protocol.DisallowedDatanodeException: Datanode denied communication with namenode because hostname cannot be resolved (ip=192.168.5.208, hostname=192.168.5.208): DatanodeRegistration(0.0.0.0:50010, datanodeUuid=ebb1de8a-6f91-4188-9f19-1240f7ebb97a, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-f2892450-2920-4136-a4de-dbdbcb5ba329;nsid=350005801;c=0)at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.registerDatanode(DatanodeManager.java:883)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerDatanode(FSNamesystem.java:4616)at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:1391)at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:100)at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:29062)at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2273)at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2269)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject.java:422)at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2267)

翻翻报错的源码

 public void registerDatanode(DatanodeRegistration nodeReg)      throws DisallowedDatanodeException, UnresolvedTopologyException {    InetAddress dnAddress = Server.getRemoteIp();    if (dnAddress != null) {      // Mostly called inside an RPC, update ip and peer hostname      String hostname = dnAddress.getHostName();      String ip = dnAddress.getHostAddress();      if (checkIpHostnameInRegistration && !isNameResolved(dnAddress)) {        // Reject registration of unresolved datanode to prevent performance        // impact of repetitive DNS lookups later.        final String message = "hostname cannot be resolved (ip="            + ip + ", hostname=" + hostname + ")";        LOG.warn("Unresolved datanode registration: " + message);        throw new DisallowedDatanodeException(nodeReg, message);      }      // update node registration with the ip and hostname from rpc request      nodeReg.setIpAddr(ip);      nodeReg.setPeerHostName(hostname);    }

从上面那个代码可以看到 错误checkIpHostnameInRegistration && !isNameResolved(dnAddress)这个判断失败进入了异常 分支

checkIpHostnameInRegistration 这个是一个配置项 通过配置dfs.namenode.datanode.registration.ip-hostname-check这个可以修改true或者false

那看一下isNameResolved这个源码

/**   * Checks if name resolution was successful for the given address.  If IP   * address and host name are the same, then it means name resolution has   * failed.  As a special case, local addresses are also considered   * acceptable.  This is particularly important on Windows, where 127.0.0.1 does   * not resolve to "localhost".   *   * @param address InetAddress to check   * @return boolean true if name resolution successful or address is local   */  private static boolean isNameResolved(InetAddress address) {    String hostname = address.getHostName();    String ip = address.getHostAddress();    return !hostname.equals(ip) || NetUtils.isLocalAddress(address);  }  

通过方法的方头注释 可以了解到 当 hostname和ip是一样的时候 返回值是false

因此进入了分支 所以我们明白了 是我们配置hdfs的时候 使用了ip代替hostname导致的错误

但是我们无法修改 所以我暂时只是修改了配置项dfs.namenode.datanode.registration.ip-hostname-check 让其不检查