01_mycat1.6源码_mycat接受客户端连接并发送握手报文

1 mycat接受客户端连接并发送握手报文

mycat1.6实现了AIO和NIO,本文只讲解NIO的流程。

在reactor模型，acceptor线程负责接受TCP连接请求。acceptor的实现类是io.mycat.net.NIOAcceptor。mycat将socket包装成前端连接FrontendConnection。

reactor线程负责处理IO请求，reactor的实现类是io.mycat.net.NIOReactor。reactor获取SelectionKey所绑定的连接AbstractConnection，调用AbstractConnection中的asynRead()和doNextWriteCheck()方法，来进行读写操作。

1.1 NIOAcceptor的实现

下面讲解上图1~3的过程。

1.1.1 NIOAcceptor的初始化

public final class NIOAcceptor extends Thread  implements SocketAcceptor{    private static final Logger LOGGER = LoggerFactory.getLogger(NIOAcceptor.class);    private static final AcceptIdGenerator ID_GENERATOR = new AcceptIdGenerator();    private final int port;    private final Selector selector;    private final ServerSocketChannel serverChannel;    private final FrontendConnectionFactory factory;    private long acceptCount;    private final NIOReactorPool reactorPool;    public NIOAcceptor(String name, String bindIp,int port,             FrontendConnectionFactory factory, NIOReactorPool reactorPool)            throws IOException {        //设置acceptor线程名称        super.setName(name);        //设置端口号        this.port = port;        //给acceptor分配选择器        this.selector = Selector.open();        //打开一个server-socket channel        this.serverChannel = ServerSocketChannel.open();        //server-socket channel设置为非阻塞        this.serverChannel.configureBlocking(false);        /** 设置TCP属性 */        //设置TCP属性,重用端口        serverChannel.setOption(StandardSocketOptions.SO_REUSEADDR, true);        //设置TCP属性,设置接收缓存        serverChannel.setOption(StandardSocketOptions.SO_RCVBUF, 1024 * 16 * 2);        // backlog=100 等待连接的最大数量        serverChannel.bind(new InetSocketAddress(bindIp, port), 100);        //注册OP_ACCEPT事件        this.serverChannel.register(selector, SelectionKey.OP_ACCEPT);        //设置连接工厂类，此处是FrontendConnectionFactory        this.factory = factory;        //设置reactorPool        this.reactorPool = reactorPool;    }

1.1.2 NIOAcceptor线程的run方法

run方法中for无限循环，selector.select(1000L)注册器设置1秒超时返回准备就绪的SelectionKey,如果SelectionKey有效且关注的是OP_ACCEPT事件，那么注册连接，否则取消SelectionKey，最后在finally块中清空所有key

@Override    public void run() {        final Selector tSelector = this.selector;        for (;;) {            ++acceptCount;            try {                tSelector.select(1000L);                Set<SelectionKey> keys = tSelector.selectedKeys();                try {                    for (SelectionKey key : keys) {                        //判断key是否有效，是否是acceptor事件                        if (key.isValid() && key.isAcceptable()) {                            //注册连接                            accept();                        } else {                            //取消key                            key.cancel();                        }                    }                } finally {                    //清空selectedKey                    keys.clear();                }            } catch (Exception e) {                LOGGER.warn(getName(), e);            }        }    }

1.1.3 NIOAcceptor的accept方法

通过accept()获取SocketChannel，设置SocketChannel为非阻塞模式，包装成前段连接FrontendConnection，设置必要参数，获取一个reactor线程，并将前段连接FrontendConnection注册到该reactor线程中进行读写操作。此处调用reactor.postRegister(c)方法只是将连接放入注册队列，后面会详细讲解。

private void accept() {        SocketChannel channel = null;        try {            //ServerSocketChannel设置成非阻塞模式。            //在非阻塞模式下，accept() 方法会立刻返回，如果还没有新进来的连接,返回的将是null。            channel = serverChannel.accept();            //channel设置成非阻塞模式            channel.configureBlocking(false);            //channel包装成前段连接            FrontendConnection c = factory.make(channel);            c.setAccepted(true);            //设置连接ID            c.setId(ID_GENERATOR.getId());            NIOProcessor processor = (NIOProcessor) MycatServer.getInstance()                    .nextProcessor();            c.setProcessor(processor);            //获取一个reactor            NIOReactor reactor = reactorPool.getNextReactor();            //注册连接            reactor.postRegister(c);        } catch (Exception e) {            LOGGER.warn(getName(), e);            closeChannel(channel);        }    }

1.1.4 NIOAcceptor的closeChannel方法

先关闭socket，再关闭channel

private static void closeChannel(SocketChannel channel) {        if (channel == null) {            return;        }        Socket socket = channel.socket();        if (socket != null) {            try {                socket.close();            } catch (IOException e) {               LOGGER.error("closeChannelError", e);            }        }        try {            channel.close();        } catch (IOException e) {            LOGGER.error("closeChannelError", e);        }    }

1.2 NIOReactor建立连接

下面讲解上图4~7的过程。

1.2.1 NIOReactor的初始化

NIOReactor初始化的时候设置一个读写线程RW,所有的IO操作实际上在这个线程中处理的。

public final class NIOReactor {    private static final Logger LOGGER = LoggerFactory.getLogger(NIOReactor.class);    private final String name;    private final RW reactorR;    public NIOReactor(String name) throws IOException {        this.name = name;        this.reactorR = new RW();    }

1.2.2 NIOReactor的启动

NIOReactor的启动实际上是启动RW线程。

    final void startup() {        new Thread(reactorR, name + "-RW").start();    }

1.2.3 RW的初始化

RW的初始化时，会获取Selector实例，并初始化一个连接注册队列。

    private final class RW implements Runnable {        private final Selector selector;        private final ConcurrentLinkedQueue<AbstractConnection> registerQueue;        private long reactCount;        private RW() throws IOException {            this.selector = Selector.open();            this.registerQueue = new ConcurrentLinkedQueue<AbstractConnection>();        }

1.2.4 RW的run方法

当selector.select(500L)方法选择一组键，其相应的通道已为 I/O 操作准备就绪，那么调用register(selector)方法（上图所示5）注册连接。如果有读写事件就绪，那么通过获取SelectionKey绑定的连接Connection，调用Connection的相应的读写方法。本文介绍首次建立连接，我们重点看register(selector)方法。

@Override        public void run() {            final Selector selector = this.selector;            Set<SelectionKey> keys = null;            for (;;) {                ++reactCount;                try {                    selector.select(500L);                    register(selector);                    keys = selector.selectedKeys();                    for (SelectionKey key : keys) {                        AbstractConnection con = null;                        try {                            Object att = key.attachment();                            if (att != null) {                                con = (AbstractConnection) att;                                if (key.isValid() && key.isReadable()) {                                    try {                                        con.asynRead();                                    } catch (IOException e) {                                        con.close("program err:" + e.toString());                                        continue;                                    } catch (Exception e) {                                        LOGGER.warn("caught err:", e);                                        con.close("program err:" + e.toString());                                        continue;                                    }                                }                                if (key.isValid() && key.isWritable()) {                                    con.doNextWriteCheck();                                }                            } else {                                key.cancel();                            }                        } catch (CancelledKeyException e) {                            if (LOGGER.isDebugEnabled()) {                                LOGGER.debug(con + " socket key canceled");                            }                        } catch (Exception e) {                            LOGGER.warn(con + " " + e);                        } catch (final Throwable e){                            // Catch exceptions such as OOM and close connection if exists                            //so that the reactor can keep running!                            // @author Uncle-pan                            // @since 2016-03-30                            if(con != null){                                con.close("Bad: "+e);                            }                            LOGGER.error("caught err: ", e);                            continue;                        }                    }                } catch (Exception e) {                    LOGGER.warn(name, e);                } catch (final Throwable e){                    // Catch exceptions such as OOM so that the reactor can keep running!                    // @author Uncle-pan                    // @since 2016-03-30                    LOGGER.error("caught err: ", e);                } finally {                    if (keys != null) {                        keys.clear();                    }                }            }        }

1.2.4 RW的register(selector)方法

当selector.select()方法获取的就绪事件或者500毫秒超时之后，那么调用register(selector)方法（上图所示5）注册连接。在register方法(上图所示6)中，循环注册队列registerQueue中的连接。

private void register(Selector selector) {            AbstractConnection c = null;            if (registerQueue.isEmpty()) {                return;            }            while ((c = registerQueue.poll()) != null) {                try {                    //注册读时间                    ((NIOSocketWR) c.getSocketWR()).register(selector);                    //向客户端发送握手报文                    c.register();                } catch (Exception e) {                    c.close("register err" + e.toString());                }            }        }

如上图7，NIOSocketWR的register(selector)方法给channel注册读事件

public void register(Selector selector) throws IOException {        try {            processKey = channel.register(selector, SelectionKey.OP_READ, con);        } finally {            if (con.isClosed.get()) {                clearSelectionKey();            }        }    }

为什么不直接在NIOAcceptor的accept()中调用NIOSocketWR的register(selector)注册读事件，而是通过注册队列的方式异步注册，主要的原因是acceptor线程直接channel.register(selector,int)方法和reactor线程的selector.select()方法会发生死锁。所以注册必须由reactor一个线程完成。

1.3 mycat向客户端发送握手报文

1.3.1 调用AbstractConnection的register方法

如上图7，调用AbstractConnection的register方法发送握手报文。

@Override    public void register() throws IOException {        if (!isClosed.get()) {            // 生成认证数据            byte[] rand1 = RandomUtil.randomBytes(8);            byte[] rand2 = RandomUtil.randomBytes(12);            // 保存认证数据            byte[] seed = new byte[rand1.length + rand2.length];            System.arraycopy(rand1, 0, seed, 0, rand1.length);            System.arraycopy(rand2, 0, seed, rand1.length, rand2.length);            this.seed = seed;            // 发送握手数据包            boolean useHandshakeV10 = MycatServer.getInstance().getConfig().getSystem().getUseHandshakeV10() == 1;            if(useHandshakeV10) {                HandshakeV10Packet hs = new HandshakeV10Packet();                hs.packetId = 0;                hs.protocolVersion = Versions.PROTOCOL_VERSION;                hs.serverVersion = Versions.SERVER_VERSION;                hs.threadId = id;                hs.seed = rand1;                hs.serverCapabilities = getServerCapabilities();                hs.serverCharsetIndex = (byte) (charsetIndex & 0xff);                hs.serverStatus = 2;                hs.restOfScrambleBuff = rand2;                hs.write(this);            } else {                HandshakePacket hs = new HandshakePacket();                hs.packetId = 0;                hs.protocolVersion = Versions.PROTOCOL_VERSION;                hs.serverVersion = Versions.SERVER_VERSION;                hs.threadId = id;                hs.seed = rand1;                hs.serverCapabilities = getServerCapabilities();                hs.serverCharsetIndex = (byte) (charsetIndex & 0xff);                hs.serverStatus = 2;                hs.restOfScrambleBuff = rand2;                hs.write(this);            }            // asynread response            this.asynRead();        }    }

1.3.2 调用HandshakePacket的write(FrontendConnection c)方法

如上图8，调用HandshakePacket的write(FrontendConnection c)方法将拼装好的握手报文写入buffer,并发送到客户端。

public void write(FrontendConnection c) {        ByteBuffer buffer = c.allocate();        BufferUtil.writeUB3(buffer, calcPacketSize());        buffer.put(packetId);        buffer.put(protocolVersion);        BufferUtil.writeWithNull(buffer, serverVersion);        BufferUtil.writeUB4(buffer, threadId);        BufferUtil.writeWithNull(buffer, seed);        BufferUtil.writeUB2(buffer, serverCapabilities);        buffer.put(serverCharsetIndex);        BufferUtil.writeUB2(buffer, serverStatus);        buffer.put(FILLER_13);        //        buffer.position(buffer.position() + 13);        BufferUtil.writeWithNull(buffer, restOfScrambleBuff);        c.write(buffer);    }

1.3.3 调用AbstractConnection的write(buffer)方法

如上图9，将buffer放入写队列并进行异步写操作。

@Override    public final void write(ByteBuffer buffer) {        if (isSupportCompress()) {            ByteBuffer newBuffer = CompressUtil.compressMysqlPacket(buffer, this, compressUnfinishedDataQueue);            writeQueue.offer(newBuffer);        } else {            writeQueue.offer(buffer);        }        // if ansyn write finishe event got lock before me ,then writing        // flag is set false but not start a write request        // so we check again        try {            this.socketWR.doNextWriteCheck();        } catch (Exception e) {            LOGGER.warn("write err:", e);            this.close("write err:" + e);        }    }

1.3.4 调用NIOSocketWR的doNextWriteCheck()方法进行异步写操作

如上图10，调用NIOSocketWR的write0()方法，把写队列中的数据写出去，如果数据没有写完，那么继续关注写事件。

public void doNextWriteCheck() {        if (!writing.compareAndSet(false, true)) {            return;        }        try {            boolean noMoreData = write0();            writing.set(false);            if (noMoreData && con.writeQueue.isEmpty()) {                if ((processKey.isValid() && (processKey.interestOps() & SelectionKey.OP_WRITE) != 0)) {                    disableWrite();                }            } else {                if ((processKey.isValid() && (processKey.interestOps() & SelectionKey.OP_WRITE) == 0)) {                    enableWrite(false);                }            }        } catch (IOException e) {            if (AbstractConnection.LOGGER.isDebugEnabled()) {                AbstractConnection.LOGGER.debug("caught err:", e);            }            con.close("err:" + e);        }    }

如果AbstractConnection的writeBuffer有数据可写，先写writeBuffer。然后再写writeQueue写队列中的数据。

private boolean write0() throws IOException {        int written = 0;        ByteBuffer buffer = con.writeBuffer;        if (buffer != null) {            while (buffer.hasRemaining()) {                written = channel.write(buffer);                if (written > 0) {                    con.netOutBytes += written;                    con.processor.addNetOutBytes(written);                    con.lastWriteTime = TimeUtil.currentTimeMillis();                } else {                    break;                }            }            if (buffer.hasRemaining()) {                con.writeAttempts++;                return false;            } else {                con.writeBuffer = null;                con.recycle(buffer);            }        }        while ((buffer = con.writeQueue.poll()) != null) {            if (buffer.limit() == 0) {                con.recycle(buffer);                con.close("quit send");                return true;            }            buffer.flip();            try {                while (buffer.hasRemaining()) {                    written = channel.write(buffer);// java.io.IOException:                                    // Connection reset by peer                    if (written > 0) {                        con.lastWriteTime = TimeUtil.currentTimeMillis();                        con.netOutBytes += written;                        con.processor.addNetOutBytes(written);                        con.lastWriteTime = TimeUtil.currentTimeMillis();                    } else {                        break;                    }                }            } catch (IOException e) {                con.recycle(buffer);                throw e;            }            if (buffer.hasRemaining()) {                con.writeBuffer = buffer;                con.writeAttempts++;                return false;            } else {                con.recycle(buffer);            }        }        return true;    }

数据写完了，取消对写事件的关注。

private void disableWrite() {        try {            SelectionKey key = this.processKey;            key.interestOps(key.interestOps() & OP_NOT_WRITE);        } catch (Exception e) {            AbstractConnection.LOGGER.warn("can't disable write " + e + " con "                    + con);        }    }

有数据需要写，关注写事件。

private void enableWrite(boolean wakeup) {        boolean needWakeup = false;        try {            SelectionKey key = this.processKey;            key.interestOps(key.interestOps() | SelectionKey.OP_WRITE);            needWakeup = true;        } catch (Exception e) {            AbstractConnection.LOGGER.warn("can't enable write " + e);        }        if (needWakeup && wakeup) {            processKey.selector().wakeup();        }    }

1.4 NIOReactor写事件处理

如果经过上面的流程数据还没有写完，那么会关注写事件，在NIOReactor类的RW线程类中在写事件就绪之后，会调用con.doNextWriteCheck()放写数据，实际上写的过程上我们是一样的，就不在重复。也是上图12~16的过程。

1.4.1 调用AbstractConnection的doNextWriteCheck()方法向客户端写数据

NIOReactor类中RW类的run方法，在写事件就绪后，调用con.doNextWriteCheck()方法写数据。

@Override        public void run() {            final Selector selector = this.selector;            Set<SelectionKey> keys = null;            for (;;) {                ++reactCount;                try {                    selector.select(500L);                    register(selector);                    keys = selector.selectedKeys();                    for (SelectionKey key : keys) {                        AbstractConnection con = null;                        try {                            Object att = key.attachment();                            if (att != null) {                                con = (AbstractConnection) att;                                if (key.isValid() && key.isReadable()) {                                    try {                                        con.asynRead();                                    } catch (IOException e) {                                        con.close("program err:" + e.toString());                                        continue;                                    } catch (Exception e) {                                        LOGGER.warn("caught err:", e);                                        con.close("program err:" + e.toString());                                        continue;                                    }                                }                                if (key.isValid() && key.isWritable()) {                                    con.doNextWriteCheck();                                }                            } else {                                key.cancel();                            }                        } catch (CancelledKeyException e) {                            if (LOGGER.isDebugEnabled()) {                                LOGGER.debug(con + " socket key canceled");                            }                        } catch (Exception e) {                            LOGGER.warn(con + " " + e);                        } catch (final Throwable e){                            // Catch exceptions such as OOM and close connection if exists                            //so that the reactor can keep running!                            // @author Uncle-pan                            // @since 2016-03-30                            if(con != null){                                con.close("Bad: "+e);                            }                            LOGGER.error("caught err: ", e);                            continue;                        }                    }                } catch (Exception e) {                    LOGGER.warn(name, e);                } catch (final Throwable e){                    // Catch exceptions such as OOM so that the reactor can keep running!                    // @author Uncle-pan                    // @since 2016-03-30                    LOGGER.error("caught err: ", e);                } finally {                    if (keys != null) {                        keys.clear();                    }                }            }        }

AbstractConnection的doNextWriteCheck()放中实际上调用的NIOSocketWR类的doNextWriteCheck()方法，过程和上图的10~12的过程。如果经过13~17的流程数据还没有写完，则继续重复13~17的流程直到数据写完为止。