ZooKeeper客户端线程模型学习

在ZooKeeper的使用过程中，大家都知道ZooKeeper客户端与服务端在建立连接时使用长连接，以此来维护客户端与服务端之间的心跳及各种命令(远程通信)，那么客户端的线程模型是什么样的呢，今天我们就来一窥究竟。

一、线程模型

ZooKeeper客户端线程模型见图一。

图一

ZooKeeper客户端与服务端通信的线程模型主要由三个队列和两个线程组成。

三个队列分别为：

1) 待发送消息队列(OutgoingQueue)

该队列主要存储需要发送的消息，使用java.util.concurrent.LinkedBlockingDeque<Packet>

2) 已发送等待响应的队列(PendingQueue)

该队列主要存储已发送的消息，使用java.util.LinkedList<Packet>

3) 事件队列(EventQueue)

该队列主要储存各种事件消息，使用java.util.concurrent.LinkedBlockingQueue<Object>

两个线程分别为：

1) 消息发送线程

该线程主要维护客户端与服务端之间的消息通信以及超时重连机制。ZooKeeper有一种状态会导致该线程退出，在与服务端建立连接时，客户端会发送连接请求报文到服务端，服务端会根据请求中信息判断该连接会话是否过期，如果该连接被服务端判定为过期，该线程会退出。

2) 事件处理线程

该线程主要处理各种事件，处理事件种类详见org.apache.zookeeper.ClientCnxn.EventThread.processEvent(Objectevent)方法。

主要实现过程

1) 同步请求

①ZooKeeper客户端实例化完成后，会同时启动消息发送线程和事件处理线程。

②客户端在提交各种操作命令时都会先封装为数据包(Packet)，加入到消息待发送队列队尾，再循环判断消息是否处理完成，未处理完成则同步等待。

③消息发送线程获取消息待发送队列队首消息，经由网络通讯模块发送消息，将发送消息在加入到已发送消息等待响应队列队尾。

④服务端返回结果，移除已发送消息等待响应队列队尾元素，处理返回结果，标记消息处理完成，并通知。

⑤客户端获取返回结果，继续执行业务处理。

2) 异步请求

①ZooKeeper客户端实例化完成后，会同时启动消息发送线程和事件处理线程。

②客户端在提交各种操作命令时都会先封装为数据包(Packet)，并提供回调函数，然后将该消息加入到消息待发送队列队尾，继续执行其它业务。

③消息发送线程从消息待发送队列获取消息，经由网络通讯模块发送消息，将发送消息在加入到已发送消息等待响应队列队尾。

④服务端返回结果，移除已发送消息等待响应队列队尾元素，处理返回结果，标记消息处理完成，并将该消息加入到事件处理队列队尾。

⑤事件处理线程获取事件处理队列队首消息，调用回调函数处理响应结果。

二、主要类图

三、源码分析

实例化ZooKeeper客户端

类名：org.apache.zookeeper. ZooKeeper

ZooKeeper客户端构造函数分别如下：

public ZooKeeper(String connectString, int sessionTimeout, Watcher watcher)  throws IOExceptionpublic ZooKeeper(String connectString, int sessionTimeout, Watcher watcher, ZKClientConfig conf) throws IOExceptionpublic ZooKeeper(String connectString, int sessionTimeout, Watcher watcher,            boolean canBeReadOnly, HostProvider aHostProvider,            ZKClientConfig clientConfig) throws IOException {        LOG.info("Initiating client connection, connectString=" + connectString                + " sessionTimeout=" + sessionTimeout + " watcher=" + watcher);        if (clientConfig == null) {            clientConfig = new ZKClientConfig();        }        this.clientConfig = clientConfig;        watchManager = defaultWatchManager();        watchManager.defaultWatcher = watcher;        ConnectStringParser connectStringParser = new ConnectStringParser(                connectString);        hostProvider = aHostProvider;        cnxn = new ClientCnxn(connectStringParser.getChrootPath(),                hostProvider, sessionTimeout, this, watchManager,                getClientCnxnSocket(), canBeReadOnly);        cnxn.start();}

其中第一个和第二个构造函数均为重载方法，其都调用第三个构造函数，并提供相应的默认的参数。

在第三个构造函数中可以看到在实例化cnxn后，立即调用了cnxn.start()方法，启动了消息发送线程和事件处理线程。

客户端连接及IO

类名：org.apache.zookeeper.ClientCnxn.SendThread

        public void run() {            ......             while (state.isAlive()) {                try {                    if (!clientCnxnSocket.isConnected()) {                        // don't re-establish connection if we are closing                        if (closing) {                            break;                        }                        startConnect();                        clientCnxnSocket.updateLastSendAndHeard();                    }                    if (state.isConnected()) {                        // determine whether we need to send an AuthFailed event.                        ...... //权限认证                        to = readTimeout - clientCnxnSocket.getIdleRecv();                    } else {                        to = connectTimeout - clientCnxnSocket.getIdleRecv();                    }                                        if (to <= 0) {                        String warnInfo;                        warnInfo = "Client session timed out, have not heard from server in "                            + clientCnxnSocket.getIdleRecv()                            + "ms"                            + " for sessionid 0x"                            + Long.toHexString(sessionId);                        LOG.warn(warnInfo);                        throw new SessionTimeoutException(warnInfo);                    }                    ......                    clientCnxnSocket.doTransport(to, pendingQueue, ClientCnxn.this);                } catch (Throwable e) {                    if (closing) {                        if (LOG.isDebugEnabled()) {                            // closing so this is expected                            LOG.debug("An exception was thrown while closing send thread for session 0x"                                    + Long.toHexString(getSessionId())                                    + " : " + e.getMessage());                        }                        break;                    } else {                        // this is ugly, you have a better way speak up                        if (e instanceof SessionExpiredException) {                            LOG.info(e.getMessage() + ", closing socket connection");                        } else if (e instanceof SessionTimeoutException) {                            LOG.info(e.getMessage() + RETRY_CONN_MSG);                        } else if (e instanceof EndOfStreamException) {                            LOG.info(e.getMessage() + RETRY_CONN_MSG);                        } else if (e instanceof RWServerFoundException) {                            LOG.info(e.getMessage());                        } else {                            LOG.warn(                                    "Session 0x"                                            + Long.toHexString(getSessionId())                                            + " for server "                                            + clientCnxnSocket.getRemoteSocketAddress()                                            + ", unexpected error"                                            + RETRY_CONN_MSG, e);                        }                        // At this point, there might still be new packets appended to outgoingQueue.                        // they will be handled in next connection or cleared up if closed.                        cleanup();                        if (state.isAlive()) {                            eventThread.queueEvent(new WatchedEvent(                                    Event.EventType.None,                                    Event.KeeperState.Disconnected,                                    null));                        }                        ......                    }                }            }            synchronized (state) {                // When it comes to this point, it guarantees that later queued                // packet to outgoingQueue will be notified of death.                cleanup();            }            clientCnxnSocket.close();            if (state.isAlive()) {                eventThread.queueEvent(new WatchedEvent(Event.EventType.None,                        Event.KeeperState.Disconnected, null));            }            ZooTrace.logTraceMessage(LOG, ZooTrace.getTextTraceLevel(),                    "SendThread exited loop for session: 0x"                           + Long.toHexString(getSessionId()));        }

可以看到while循环中state.isAlive()返回true则该线程就不会退出，state.isAlive()该方法判断只要连接状态不是关闭或者权限验证失败就返回true，对应方法详见org.apache.zookeeper.ZooKeeper.States类。

该方法中有三个超时时间需要注意：

1) 会话超时时间(SessionTimeout)：当首次进行连接时，会话超时时间为配置参数传入时间。

2) 连接超时时间：当首次进行连接时，连接超时时间为会话超时间/客户端连接串返回服务端个(sessionTimeout / hostProvider.size())；当连接建立后和服务端通信，返回协商时间negotiatedSessionTimeout,连接超时时间为negotiatedSessionTimeout/客户端连接串返回服务端个数(connectTimeout= negotiatedSessionTimeout / hostProvider.size())

3) 读超时时间：当首次进行连接时读超时时间为会话超时时间*2/3，设置公式为sessionTimeout * 2 / 3；当连接建立后和服务端通信，返回协商时间negotiatedSessionTimeout,设置公式为negotiatedSessionTimeout* 2 / 3

4) Session超时判断：当是在连接时，判断条件为连接超时时间-通信空闲时间(当前时间-上次消息发送时间)<0为超时，当是在连接建立后，通信时，判断条件为读超时时间-通信空闲时间<0为超时，会引起Session超时异常。

 

类名：org.apache.zookeeper.ClientCnxnSocketNIO

void registerAndConnect(SocketChannel sock, InetSocketAddress addr) throws IOException {        sockKey = sock.register(selector, SelectionKey.OP_CONNECT);        boolean immediateConnect = sock.connect(addr); //如果返回true表示连接已经建立完成，返回false如果要处理连接建立完成事件，需要关注sockKey对应的状态。        if (immediateConnect) {            sendThread.primeConnection();        }    }void doTransport(int waitTimeOut, List<Packet> pendingQueue, ClientCnxn cnxn)            throws IOException, InterruptedException {        selector.select(waitTimeOut);        Set<SelectionKey> selected;        synchronized (this) {            selected = selector.selectedKeys();        }        // Everything below and until we get back to the select is        // non blocking, so time is effectively a constant. That is        // Why we just have to do this once, here        updateNow();        for (SelectionKey k : selected) {            SocketChannel sc = ((SocketChannel) k.channel());            if ((k.readyOps() & SelectionKey.OP_CONNECT) != 0) {                if (sc.finishConnect()) {   //连接建立完成后,发送连接数据包                    updateLastSendAndHeard();                    updateSocketAddresses();                    sendThread.primeConnection();                }            } else if ((k.readyOps() & (SelectionKey.OP_READ | SelectionKey.OP_WRITE)) != 0) {                doIO(pendingQueue, cnxn);            }        }        if (sendThread.getZkState().isConnected()) {            if (findSendablePacket(outgoingQueue,                    sendThread.tunnelAuthInProgress()) != null) {                enableWrite();            }        }        selected.clear();    }void doIO(List<Packet> pendingQueue, ClientCnxn cnxn) throws InterruptedException, IOException {        SocketChannel sock = (SocketChannel) sockKey.channel();        if (sock == null) {            throw new IOException("Socket is null!");        }        if (sockKey.isReadable()) {            int rc = sock.read(incomingBuffer);            if (rc < 0) {                throw new EndOfStreamException(                        "Unable to read additional data from server sessionid 0x"                                + Long.toHexString(sessionId)                                + ", likely server has closed socket");            }            if (!incomingBuffer.hasRemaining()) {                incomingBuffer.flip();                if (incomingBuffer == lenBuffer) {                    recvCount++;                    readLength();                } else if (!initialized) {                    readConnectResult();                    enableRead();                    if (findSendablePacket(outgoingQueue,                            sendThread.tunnelAuthInProgress()) != null) {                        // Since SASL authentication has completed (if client is configured to do so),                        // outgoing packets waiting in the outgoingQueue can now be sent.                        enableWrite();                    }                    lenBuffer.clear();                    incomingBuffer = lenBuffer;                    updateLastHeard();                    initialized = true;                } else {                    sendThread.readResponse(incomingBuffer);                    lenBuffer.clear();                    incomingBuffer = lenBuffer;                    updateLastHeard();                }            }        }        ......}

这段IO代码比较有意思，在读取数据流操作时，首先判断incomingBuffer == lenBuffer，然而lenBuffer为ClientCnxnSocket中一个受保护的成员变量，其修饰符为final，当该条件成立时，读取数据流长度，并为incomingBuffer重新分配对应长度的数据数组，在下次数据读取时，读取对应长度的消息报文，再调用sendThread.readResponse(incomingBuffer)解析数据，继续清除数据，并将lenBuffer再次赋值给incomingBuffer。当第一次读取了数据，而initialized为false时，该时候读取到的数据包为建立连接后服务端推送的数据包，该数据包返回值会确定客户端连接的协商超时事件，会话ID等选项。

registerAndConnect方法是在发送消息线程中run方法中调用startConnect()调用。在客户端与服务端连接建立后会首先调用sendThread.primeConnection方法发送连接请求数据包。

创建节点

类名：org.apache.zookeeper. ZooKeeper

 public String create(final String path, byte data[], List<ACL> acl, CreateMode createMode) throws KeeperException, InterruptedException    {        final String clientPath = path;        PathUtils.validatePath(clientPath, createMode.isSequential());        EphemeralType.validateTTL(createMode, -1);        final String serverPath = prependChroot(clientPath);        RequestHeader h = new RequestHeader();        h.setType(createMode.isContainer() ? ZooDefs.OpCode.createContainer : ZooDefs.OpCode.create);        CreateRequest request = new CreateRequest();        CreateResponse response = new CreateResponse();        request.setData(data);        request.setFlags(createMode.toFlag());        request.setPath(serverPath);        if (acl != null && acl.size() == 0) {            throw new KeeperException.InvalidACLException();        }        request.setAcl(acl);        ReplyHeader r = cnxn.submitRequest(h, request, response, null);        if (r.getErr() != 0) {            throw KeeperException.create(KeeperException.Code.get(r.getErr()),                    clientPath);        }        if (cnxn.chrootPath == null) {            return response.getPath();        } else {            return response.getPath().substring(cnxn.chrootPath.length());        }}public void create(final String path, byte data[], List<ACL> acl, CreateMode createMode, StringCallback cb, Object ctx)    {        final String clientPath = path;        PathUtils.validatePath(clientPath, createMode.isSequential());        EphemeralType.validateTTL(createMode, -1);        final String serverPath = prependChroot(clientPath);        RequestHeader h = new RequestHeader();        h.setType(createMode.isContainer() ? ZooDefs.OpCode.createContainer : ZooDefs.OpCode.create);        CreateRequest request = new CreateRequest();        CreateResponse response = new CreateResponse();        ReplyHeader r = new ReplyHeader();        request.setData(data);        request.setFlags(createMode.toFlag());        request.setPath(serverPath);        request.setAcl(acl);        cnxn.queuePacket(h, r, request, response, cb, clientPath,                serverPath, ctx, null);}

类名：org.apache.zookeeper.ClientCnxn

    public ReplyHeader submitRequest(RequestHeader h, Record request, Record response, WatchRegistration watchRegistration)            throws InterruptedException {        return submitRequest(h, request, response, watchRegistration, null);    }    public ReplyHeader submitRequest(RequestHeader h, Record request, Record response, WatchRegistration watchRegistration,            WatchDeregistration watchDeregistration)            throws InterruptedException {        ReplyHeader r = new ReplyHeader();        Packet packet = queuePacket(h, r, request, response, null, null, null,                null, watchRegistration, watchDeregistration);        synchronized (packet) {            while (!packet.finished) {                packet.wait();            }        }        return r;    }    public Packet queuePacket(RequestHeader h, ReplyHeader r, Record request, Record response, AsyncCallback cb, String clientPath,            String serverPath, Object ctx, WatchRegistration watchRegistration) {        return queuePacket(h, r, request, response, cb, clientPath, serverPath,                ctx, watchRegistration, null);    }    public Packet queuePacket(RequestHeader h, ReplyHeader r, Record request, Record response, AsyncCallback cb, String clientPath,            String serverPath, Object ctx, WatchRegistration watchRegistration,            WatchDeregistration watchDeregistration) {        Packet packet = null;        // Note that we do not generate the Xid for the packet yet. It is        // generated later at send-time, by an implementation of ClientCnxnSocket::doIO(),        // where the packet is actually sent.        packet = new Packet(h, r, request, response, watchRegistration);        packet.cb = cb;        packet.ctx = ctx;        packet.clientPath = clientPath;        packet.serverPath = serverPath;        packet.watchDeregistration = watchDeregistration;        // The synchronized block here is for two purpose:        // 1. synchronize with the final cleanup() in SendThread.run() to avoid race        // 2. synchronized against each packet. So if a closeSession packet is added,        // later packet will be notified.        synchronized (state) {            if (!state.isAlive() || closing) {                conLossPacket(packet);            } else {                // If the client is asking to close the session then                // mark as closing                if (h.getType() == OpCode.closeSession) {                    closing = true;                }                outgoingQueue.add(packet);            }        }        sendThread.getClientCnxnSocket().packetAdded();        return packet;    }    private void finishPacket(Packet p) {        ......        if (p.cb == null) {            synchronized (p) {                p.finished = true;                p.notifyAll();            }        } else {            p.finished = true;            eventThread.queuePacket(p);        }    }

类名：org.apache.zookeeper.ClientCnxn.SendThread

     void readResponse(ByteBuffer incomingBuffer) throws IOException {            ByteBufferInputStream bbis = new ByteBufferInputStream(                    incomingBuffer);            BinaryInputArchive bbia = BinaryInputArchive.getArchive(bbis);            ReplyHeader replyHdr = new ReplyHeader();            replyHdr.deserialize(bbia, "header");            ......            Packet packet;            synchronized (pendingQueue) {                if (pendingQueue.size() == 0) {                    throw new IOException("Nothing in the queue, but got "                            + replyHdr.getXid());                }                packet = pendingQueue.remove();            }            /*             * Since requests are processed in order, we better get a response             * to the first request!             */            try {                if (packet.requestHeader.getXid() != replyHdr.getXid()) {                    packet.replyHeader.setErr(                            KeeperException.Code.CONNECTIONLOSS.intValue());                    throw new IOException("Xid out of order. Got Xid "                            + replyHdr.getXid() + " with err " +                            + replyHdr.getErr() +                            " expected Xid "                            + packet.requestHeader.getXid()                            + " for a packet with details: "                            + packet );                }                packet.replyHeader.setXid(replyHdr.getXid());                packet.replyHeader.setErr(replyHdr.getErr());                packet.replyHeader.setZxid(replyHdr.getZxid());                if (replyHdr.getZxid() > 0) {                    lastZxid = replyHdr.getZxid();                }                if (packet.response != null && replyHdr.getErr() == 0) {                    packet.response.deserialize(bbia, "response");                }                if (LOG.isDebugEnabled()) {                    LOG.debug("Reading reply sessionid:0x"                            + Long.toHexString(sessionId) + ", packet:: " + packet);                }            } finally {                finishPacket(packet);            }        }

类名：org.apache.zookeeper.ClientCnxn.EventThread

      private void processEvent(Object event) {          try {              ......                  Packet p = (Packet) event;                  int rc = 0;                  String clientPath = p.clientPath;                  if (p.replyHeader.getErr() != 0) {                      rc = p.replyHeader.getErr();                  }                  if (p.cb == null) {                      LOG.warn("Somehow a null cb got to EventThread!");                  } else if (p.response instanceof ExistsResponse                          || p.response instanceof SetDataResponse                          || p.response instanceof SetACLResponse) {                      StatCallback cb = (StatCallback) p.cb;                      if (rc == 0) {                          if (p.response instanceof ExistsResponse) {                              cb.processResult(rc, clientPath, p.ctx,                                      ((ExistsResponse) p.response)                                              .getStat());                          } else if (p.response instanceof SetDataResponse) {                              cb.processResult(rc, clientPath, p.ctx,                                      ((SetDataResponse) p.response)                                              .getStat());                          } else if (p.response instanceof SetACLResponse) {                              cb.processResult(rc, clientPath, p.ctx,                                      ((SetACLResponse) p.response)                                              .getStat());                          }                      } else {                          cb.processResult(rc, clientPath, p.ctx, null);                      }                  }               ......              }          } catch (Throwable t) {              LOG.error("Caught unexpected throwable", t);          }       }    }

可以看到ZooKeeper客户端在提交请求时，同步创建节点调用了cnxn.submitRequest(h, request, response, null)方法，异步创建节点直接调用了cnxn.queuePacket方法。

在同步请求中，submitRequest其实也是对queuePacket进行了包装，让其达到同步的效果。

执行操作时，queuePacket方法先将请求封装为Packet加入到消息待发送队列，然后在cnxn.submitRequest方法中同步Packet对象，循环调用packet.finished判断消息是否处理完成，没有则等待packet.wait()方法等待消息处理完成。IO线程发送消息并在接收到服务端响应后，调用readResponse方法，该方法对返回结果进行反序列化处理，最终并调用finishPacket方法。在finishPacket方法中，可以看到，如果p.cb如果为空的话，则表示同步请求，则调用packet.notifyAll方法通知等待该响应结果的线程；如果p.cb不为空，则将数据包加入到eventQueue中，由事件处理线程调用processEvent处理事件。