HDFS的文件操作流(4)——写操作(数据节点)

来源：互联网发布：linux基础教程编辑：程序博客网时间：2024/05/21 17:35

上一篇本文我详细的分析了在HDFS的文件写操作中，客户端是如何工作的，其工作核心可总结为两点：一是向NameNode申请Block，二是向数据节点传输Block的packet。那么，数据节点是如何来接受这个数据块的呢？这个还得从数据节点的注册说起。

数据节点在启动之后，会向NameNode节点进行注册来告诉它自己的一些信息，这些信息包括自己的存储信息，服务信息等。其中，服务信息包括自己的数据接受地址、RPC地址、Info地址。数据接受地址就是用来接受客户端发送过来的有关数据块的操作。而这个工作统一交给DataXceiverServer来管理，这个DataXceiverServer实际上可以看做是一个线程池，具体的为某一个客户端服务的话，它还是交给工作线程DataXceiver来做的。好吧，那就来具体的看看DataXceiver。

class DataXceiver implements Runnable, FSConstants {    public void run() {          DataInputStream in=null;    try {      in = new DataInputStream( new BufferedInputStream(NetUtils.getInputStream(s), SMALL_BUFFER_SIZE));      short version = in.readShort();      if ( version != DataTransferProtocol.DATA_TRANSFER_VERSION ) {        throw new IOException( "Version Mismatch" );      }          boolean local = s.getInetAddress().equals(s.getLocalAddress());      byte op = in.readByte();            // Make sure the xciver count is not exceeded      int curXceiverCount = datanode.getXceiverCount();      if (curXceiverCount > dataXceiverServer.maxXceiverCount) {        throw new IOException("xceiverCount " + curXceiverCount  + " exceeds the limit of concurrent xcievers " + dataXceiverServer.maxXceiverCount);      }            long startTime = DataNode.now();      switch ( op ) {      case DataTransferProtocol.OP_READ_BLOCK:        readBlock( in );        datanode.myMetrics.readBlockOp.inc(DataNode.now() - startTime);        if (local)          datanode.myMetrics.readsFromLocalClient.inc();        else          datanode.myMetrics.readsFromRemoteClient.inc();        break;      case DataTransferProtocol.OP_WRITE_BLOCK:        writeBlock( in );        datanode.myMetrics.writeBlockOp.inc(DataNode.now() - startTime);        if (local)          datanode.myMetrics.writesFromLocalClient.inc();        else          datanode.myMetrics.writesFromRemoteClient.inc();        break;      case DataTransferProtocol.OP_READ_METADATA:        readMetadata( in );        datanode.myMetrics.readMetadataOp.inc(DataNode.now() - startTime);        break;      case DataTransferProtocol.OP_REPLACE_BLOCK: // for balancing purpose; send to a destination        replaceBlock(in);        datanode.myMetrics.replaceBlockOp.inc(DataNode.now() - startTime);        break;      case DataTransferProtocol.OP_COPY_BLOCK:            // for balancing purpose; send to a proxy source        copyBlock(in);        datanode.myMetrics.copyBlockOp.inc(DataNode.now() - startTime);        break;      case DataTransferProtocol.OP_BLOCK_CHECKSUM: //get the checksum of a block        getBlockChecksum(in);        datanode.myMetrics.blockChecksumOp.inc(DataNode.now() - startTime);        break;      default:        throw new IOException("Unknown opcode " + op + " in data stream");      }    } catch (Throwable t) {      LOG.error(datanode.dnRegistration + ":DataXceiver",t);    } finally {      LOG.debug(datanode.dnRegistration + ":Number of active connections is: " + datanode.getXceiverCount());      IOUtils.closeStream(in);      IOUtils.closeSocket(s);      dataXceiverServer.childSockets.remove(s);    }  }}

在DataXceiver的run()方法中，他根据客户端的操作要求(如：读数据块、写数据块、复制数据块等)来调用响应的函数处理，本文既是谈论文件的写操作，那么自然就会具体分析writeBlock()方法。

private void writeBlock(DataInputStream in) throws IOException {    DatanodeInfo srcDataNode = null;    LOG.debug("writeBlock receive buf size " + s.getReceiveBufferSize() + " tcp no delay " + s.getTcpNoDelay());    //先取出相关的头部信息     Block block = new Block(in.readLong(), dataXceiverServer.estimateBlockSize, in.readLong());//Block的id和创建时间      int pipelineSize = in.readInt(); //Block副本数量    boolean isRecovery = in.readBoolean(); // 是否是恢复Block    String client = Text.readString(in); //客户端名字    boolean hasSrcDataNode = in.readBoolean(); // 是否来自数据节点    if (hasSrcDataNode) {      srcDataNode = new DatanodeInfo();      srcDataNode.readFields(in);//源数据节点信息    }    int numTargets = in.readInt();//存放剩余Block副本数量    if (numTargets < 0) {      throw new IOException("Mislabelled incoming datastream.");    }    DatanodeInfo targets[] = new DatanodeInfo[numTargets];    for (int i = 0; i < targets.length; i++) {      DatanodeInfo tmp = new DatanodeInfo();      tmp.readFields(in);//存放剩余Block副本的数据节点信息      targets[i] = tmp;    }    DataOutputStream mirrorOut = null;  // 向下一个数据节点写入Block的网络I/O流    DataInputStream mirrorIn = null;    // reply from next target        DataOutputStream replyOut = null;   // stream to prev target        Socket mirrorSock = null;           // socket to next target    BlockReceiver blockReceiver = null; // responsible for data handling    String mirrorNode = null;           // the name:port of next target    String firstBadLink = "";           // first datanode that failed in connection setup    try {           blockReceiver = new BlockReceiver(block, in, s.getRemoteSocketAddress().toString(), s.getLocalSocketAddress().toString(), isRecovery, client, srcDataNode, datanode);      if (targets.length > 0) {        InetSocketAddress mirrorTarget = null;        // Connect to backup machine        mirrorNode = targets[0].getName();        mirrorTarget = NetUtils.createSocketAddr(mirrorNode);        mirrorSock = datanode.newSocket();        try {          int timeoutValue = numTargets * datanode.socketTimeout;          int writeTimeout = datanode.socketWriteTimeout + (HdfsConstants.WRITE_TIMEOUT_EXTENSION * numTargets);          NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue);          mirrorSock.setSoTimeout(timeoutValue);          mirrorSock.setSendBufferSize(DEFAULT_DATA_SOCKET_SIZE);          mirrorOut = new DataOutputStream(new BufferedOutputStream(NetUtils.getOutputStream(mirrorSock, writeTimeout),SMALL_BUFFER_SIZE));          mirrorIn = new DataInputStream(NetUtils.getInputStream(mirrorSock));          // 向下一个数据节点写入Block的头部信息          mirrorOut.writeShort( DataTransferProtocol.DATA_TRANSFER_VERSION );          mirrorOut.write( DataTransferProtocol.OP_WRITE_BLOCK );          mirrorOut.writeLong( block.getBlockId() );          mirrorOut.writeLong( block.getGenerationStamp() );          mirrorOut.writeInt( pipelineSize );          mirrorOut.writeBoolean( isRecovery );          Text.writeString( mirrorOut, client );          mirrorOut.writeBoolean(hasSrcDataNode);          if (hasSrcDataNode) { // pass src node information            srcDataNode.write(mirrorOut);          }          mirrorOut.writeInt( targets.length - 1 );          for ( int i = 1; i < targets.length; i++ ) {            targets[i].write( mirrorOut );          }          //向下一个数据节点写入校验信息          blockReceiver.writeChecksumHeader(mirrorOut);          mirrorOut.flush();          //获取下一个数据节点的确认信息          if (client.length() != 0) {            firstBadLink = Text.readString(mirrorIn);            if (LOG.isDebugEnabled() || firstBadLink.length() > 0) {              LOG.info("Datanode " + targets.length +                       " got response for connect ack " +                       " from downstream datanode with firstbadlink as " +                       firstBadLink);            }          }        } catch (IOException e) {          if (client.length() != 0) {            Text.writeString(replyOut, mirrorNode);            replyOut.flush();          }          IOUtils.closeStream(mirrorOut);          mirrorOut = null;          IOUtils.closeStream(mirrorIn);          mirrorIn = null;          IOUtils.closeSocket(mirrorSock);          mirrorSock = null;          if (client.length() > 0) {            throw e;          } else {          }        }      }      if (client.length() != 0) {        if (LOG.isDebugEnabled() || firstBadLink.length() > 0) {        }        Text.writeString(replyOut, firstBadLink);//向前一个一个数据节点或客户回复        replyOut.flush();      }      // 接受前一个数据节点或客户端发送来的Block，并把该Block发送到下一个数据节点      String mirrorAddr = (mirrorSock == null) ? null : mirrorNode;      blockReceiver.receiveBlock(mirrorOut, mirrorIn, replyOut, mirrorAddr, null, targets.length);      if (client.length() == 0) {        //将接收到的一个Block添加到当前数据节点的receivedBlockList        datanode.notifyNamenodeReceivedBlock(block, DataNode.EMPTY_DEL_HINT);          if (datanode.blockScanner != null) {        datanode.blockScanner.addBlock(block);      }          } catch (IOException ioe) {      throw ioe;    } finally {      // close all opened streams      IOUtils.closeStream(mirrorOut);      IOUtils.closeStream(mirrorIn);      IOUtils.closeStream(replyOut);      IOUtils.closeSocket(mirrorSock);      IOUtils.closeStream(blockReceiver);    }  }

在writeBlock()方法中，数据节点主要是靠BlockReceiver来接受前一个客户端或者是数据接节点发送过来的Block，并把它发送到下一个数据节点。关于BlockReceiver的实现细节在这里我就不再赘述了，有兴趣的盆友可以自己去研究一下它的源代码。当一个DataXceiver成功地接受完一个Block并把它发送到下一个数据节点之后，它就会把刚收到的一个Block放到DataNode的receivedBlockList队列中，而DataNode也会将该Block报告给NameNode节点。

另外，我还用补充一个问题就是关于数据节点的数据传输服务地址的问题。我曾经遇到过这样一个有趣的问题：我只在一台pc上部署HDFS，一个NameNode节点，一个DataNode节点，其中我的pc的ip地址是*.*.*.*(教育网ip地址)，然后我把NameNode的服务地址配置成localhost:8020，DataNode的数据服务地址配置成*.*.*.*:50010，结果客户端在向数据节点传输Block时始终都无法连接到DataNode的数据服务端口。盆友们知道这是为什么吗？这是因为数据节点在向NameNode节点注册时，NameNode执行了下面一段代码：

String dnAddress = Server.getRemoteAddress();    if (dnAddress == null) {      dnAddress = nodeReg.getHost();    }          // check if the datanode is allowed to be connect to the namenode    if (!verifyNodeRegistration(nodeReg, dnAddress)) {      throw new DisallowedDatanodeException(nodeReg);    }    String hostName = nodeReg.getHost();          // update the datanode's name with ip:port    DatanodeID dnReg = new DatanodeID(dnAddress + ":" + nodeReg.getPort(),  nodeReg.getStorageID(),  nodeReg.getInfoPort(), nodeReg.getIpcPort());    nodeReg.updateRegInfo(dnReg);

在上面的代码中，NameNode节点一开始就根据连接的数据节点的远程地址更新了注册信息nodeReg，即修改了数据节点的数据服务地址，问题也就出在这儿了。由于我设置的NameNode的地址是localhost:8020，所以当数据节点向NameNode节点连接时，并没有走路由器，那么NameNode收到的数据节点连接的远程地址就是127.0.0.1了，即Server.getRemoteAddress()会返回127.0.0.1，客户端收到的数据节点的数据服务地址就是127.0.0.1:50010，同时数据节点也干了一件轰动的事情，就是把数据服务地址绑定到了*.*.*.*:50010上，而不是只绑定到某一个端口上，着一系列的动作就导致了客户端找不到数据节点的数据服务地址。之后，我就索性直接把NameNode的地址配置成*.*.*.*:8020，然后半点事没有。