cobar源码浅析

网路连接部分

网络连接部分是cobar非常重要的部分，它承接着前向连接（面向用户）和后向连接（面向mysql）。通过分析网络连接代码，可以大致看出cobar的数据流向。

我们先看看整个连接架构（省略心跳相关内容）：

我们再来看看系统初始化代码

        // startup processors        LOGGER.info("Startup processors ...");        int handler = system.getProcessorHandler();        int executor = system.getProcessorExecutor();        int committer = system.getProcessorCommitter();        processors = new NIOProcessor[system.getProcessors()];        for (int i = 0; i < processors.length; i++) {            processors[i] = new NIOProcessor("Processor" + i, handler, executor, committer);            processors[i].startup();        }        timer.schedule(processorCheck(), 0L, system.getProcessorCheckPeriod());        // startup connector        LOGGER.info("Startup connector ...");        connector = new NIOConnector(NAME + "Connector");        connector.setProcessors(processors);        connector.start();        // init dataNodes        Map<String, MySQLDataNode> dataNodes = config.getDataNodes();        LOGGER.info("Initialize dataNodes ...");        for (MySQLDataNode node : dataNodes.values()) {            node.init(1, 0);        }        timer.schedule(dataNodeIdleCheck(), 0L, system.getDataNodeIdleCheckPeriod());        timer.schedule(dataNodeHeartbeat(), 0L, system.getDataNodeHeartbeatPeriod());        // startup manager        ManagerConnectionFactory mf = new ManagerConnectionFactory();        mf.setCharset(system.getCharset());        mf.setIdleTimeout(system.getIdleTimeout());        manager = new NIOAcceptor(NAME + "Manager", system.getManagerPort(), mf);        manager.setProcessors(processors);        manager.start();        LOGGER.info(manager.getName() + " is started and listening on " + manager.getPort());        // startup server        ServerConnectionFactory sf = new ServerConnectionFactory();        sf.setCharset(system.getCharset());        sf.setIdleTimeout(system.getIdleTimeout());        server = new NIOAcceptor(NAME + "Server", system.getServerPort(), sf);        server.setProcessors(processors);        server.start();        timer.schedule(clusterHeartbeat(), 0L, system.getClusterHeartbeatPeriod());

从代码可以看出，系统首先根据系统内核数量，初始化相应数量的nio处理器，然后初始化连接器，节点初始化，管理线程启动，服务线程启动。可以看到，管理线程和服务线程共享处理器。此外就是一些定时器了，这里先暂时不讲解。管理器是内部人员管理使用，这里我们也先搁置。剩下NIOProcessor, MySQLDataNode, NIOAcceptor。 因为NIOProcessor是共用产品，我们先看看它的结构。

NIOProcessor

    private final String name;    private final NIOReactor reactor;    private final BufferPool bufferPool;    private final NameableExecutor handler;    private final NameableExecutor executor;    private final NameableExecutor committer;    private final ConcurrentMap<Long, FrontendConnection> frontends;    private final ConcurrentMap<Long, BackendConnection> backends;    private final CommandCount commands;    private long netInBytes;    private long netOutBytes;

以上为NIOProcessor的字段信息，其中较为重要的为NIOReactor ， BufferPool。 BufferPool是一个缓冲池，这里不多讲；我们看看NIOReactor 的内容

    private final R reactorR;    private final W reactorW;    public NIOReactor(String name) throws IOException {        this.name = name;        this.reactorR = new R();        this.reactorW = new W();    }    final void startup() {        new Thread(reactorR, name + "-R").start();        new Thread(reactorW, name + "-W").start();    }    final void postRegister(NIOConnection c) {        reactorR.registerQueue.offer(c);        reactorR.selector.wakeup();    }    final void postWrite(NIOConnection c) {        reactorW.writeQueue.offer(c);    }

在reactor里面维护了两个线程，每个线程分配一个blockingqueue。R线程负责从队列中读取连接进行读操作或者写操作，W线程负责写操作。

NIOAcceptor

NIOAcceptor的主要功能是根据指定的端口，接收用户执行的mysql语句，轮询选择NIOProcessor进行处理，我们看一看它的accept代码。首先接收channel，利用工厂（ServerConnectionFactory）创建FrontendConnection 。

private void accept() {        SocketChannel channel = null;        try {            channel = serverChannel.accept();            channel.configureBlocking(false);            FrontendConnection c = factory.make(channel);            c.setAccepted(true);            c.setId(ID_GENERATOR.getId());            NIOProcessor processor = nextProcessor();            c.setProcessor(processor);            processor.postRegister(c);        } catch (Throwable e) {            closeChannel(channel);            LOGGER.warn(getName(), e);        }    }    protected FrontendConnection getConnection(SocketChannel channel) {        SystemConfig sys = CobarServer.getInstance().getConfig().getSystem();        ServerConnection c = new ServerConnection(channel);        c.setPrivileges(new CobarPrivileges());        c.setQueryHandler(new ServerQueryHandler(c));        // c.setPrepareHandler(new ServerPrepareHandler(c)); TODO prepare        c.setTxIsolation(sys.getTxIsolation());        c.setSession(new BlockingSession(c));        c.setSession2(new NonBlockingSession(c));        return c;    }

通过代码可以知道，对每一个请求，cobar会创建一个ServerConnection，选择一个NIOProcessor 处理，ServerConnection就是FrontendConnection 的一个子类实现。接下来我们具体分析下ServerConnection。

前向连接
前向连接主要实现是ServerConnection（这里不包括管理部分）。 我们看到accept()函数里面有processor.postRegister(c)语句，我们看看这条语句做了什么事

    将连接提交到队列中    final void postRegister(NIOConnection c) {        reactorR.registerQueue.offer(c);        reactorR.selector.wakeup();    }    注册事件类型，注意到这里将key保存起来了    public void register(Selector selector) throws IOException {        try {            processKey = channel.register(selector, SelectionKey.OP_READ, this);            isRegistered = true;        } finally {            if (isClosed.get()) {                clearSelectionKey();            }        }    }    读数据进行处理    public void read() throws IOException {        ByteBuffer buffer = this.readBuffer;        int got = channel.read(buffer);        lastReadTime = TimeUtil.currentTimeMillis();        if (got < 0) {            throw new EOFException();        }        netInBytes += got;        processor.addNetInBytes(got);        int offset = readBufferOffset, length = 0, position = buffer.position();        for (;;) {            length = getPacketLength(buffer, offset);            if (length == -1) {                if (!buffer.hasRemaining()) {                    checkReadBuffer(buffer, offset, position);                }                break;            }            if (position >= offset + length) {                buffer.position(offset);                byte[] data = new byte[length];                buffer.get(data, 0, length);                handle(data);                offset += length;                if (position == offset) {                    if (readBufferOffset != 0) {                        readBufferOffset = 0;                    }                    buffer.clear();                    break;                } else {                    readBufferOffset = offset;                    buffer.position(position);                    continue;                }            } else {                if (!buffer.hasRemaining()) {                    checkReadBuffer(buffer, offset, position);                }                break;            }        }    }    重置命令处理类型    source.setHandler(new FrontendCommandHandler(source));    ByteBuffer buffer = source.allocate();    source.write(source.writeToBuffer(AUTH_OK, buffer));    处理数据    public void handle(final byte[] data) {        这里异步处理前端数据        processor.getHandler().execute(new Runnable() {            @Override            public void run() {                try {                    handler.handle(data);                } catch (Throwable t) {                    error(ErrorCode.ERR_HANDLE_DATA, t);                }            }        });    }    回复请求    public void write(ByteBuffer buffer) {        if (isClosed.get()) {            processor.getBufferPool().recycle(buffer);            return;        }        if (isRegistered) {            try {                writeQueue.put(buffer);            } catch (InterruptedException e) {                error(ErrorCode.ERR_PUT_WRITE_QUEUE, e);                return;            }            processor.postWrite(this);        } else {            processor.getBufferPool().recycle(buffer);            close();        }    }    private void enableWrite() {        final Lock lock = this.keyLock;        lock.lock();        try {            SelectionKey key = this.processKey;            key.interestOps(key.interestOps() | SelectionKey.OP_WRITE);        } finally {            lock.unlock();        }        processKey.selector().wakeup();    }    private void write(NIOConnection c) {        try {            c.writeByQueue();        } catch (Throwable e) {            c.error(ErrorCode.ERR_WRITE_BY_QUEUE, e);        }    }    处理数据    public void handle(byte[] data) {        switch (data[4]) {        case MySQLPacket.COM_INIT_DB:            commands.doInitDB();            source.initDB(data);            break;        case MySQLPacket.COM_QUERY:            commands.doQuery();            source.query(data);            break;        case MySQLPacket.COM_PING:            commands.doPing();            source.ping();            break;        case MySQLPacket.COM_QUIT:            commands.doQuit();            source.close();            break;        case MySQLPacket.COM_PROCESS_KILL:            commands.doKill();            source.kill(data);            break;        case MySQLPacket.COM_STMT_PREPARE:            commands.doStmtPrepare();            source.stmtPrepare(data);            break;        case MySQLPacket.COM_STMT_EXECUTE:            commands.doStmtExecute();            source.stmtExecute(data);            break;        case MySQLPacket.COM_STMT_CLOSE:            commands.doStmtClose();            source.stmtClose(data);            break;        case MySQLPacket.COM_HEARTBEAT:            commands.doHeartbeat();            source.heartbeat(data);            break;        default:            commands.doOther();            source.writeErrMessage(ErrorCode.ER_UNKNOWN_COM_ERROR, "Unknown command");        }    }    以query为例，我们看看往下怎么进行    public void query(byte[] data) {        if (queryHandler != null) {            MySQLMessage mm = new MySQLMessage(data);            mm.position(5);            String sql = null;            try {                sql = mm.readString(charset);            } catch (UnsupportedEncodingException e) {                writeErrMessage(ErrorCode.ER_UNKNOWN_CHARACTER_SET, "Unknown charset '" + charset + "'");                return;            }            if (sql == null || sql.length() == 0) {                writeErrMessage(ErrorCode.ER_NOT_ALLOWED_COMMAND, "Empty SQL");                return;            }            queryHandler.query(sql);        } else {            writeErrMessage(ErrorCode.ER_UNKNOWN_COM_ERROR, "Query unsupported!");        }    }    @Override    public void query(String sql) {        ServerConnection c = this.source;        if (LOGGER.isDebugEnabled()) {            LOGGER.debug(new StringBuilder().append(c).append(sql).toString());        }        int rs = ServerParse.parse(sql);        switch (rs & 0xff) {        case ServerParse.EXPLAIN:            ExplainHandler.handle(sql, c, rs >>> 8);            break;        case ServerParse.SET:            SetHandler.handle(sql, c, rs >>> 8);            break;        case ServerParse.SHOW:            ShowHandler.handle(sql, c, rs >>> 8);            break;        case ServerParse.SELECT:            SelectHandler.handle(sql, c, rs >>> 8);            break;        case ServerParse.START:            StartHandler.handle(sql, c, rs >>> 8);            break;        case ServerParse.BEGIN:            BeginHandler.handle(sql, c);            break;        case ServerParse.SAVEPOINT:            SavepointHandler.handle(sql, c);            break;        case ServerParse.KILL:            KillHandler.handle(sql, rs >>> 8, c);            break;        case ServerParse.KILL_QUERY:            c.writeErrMessage(ErrorCode.ER_UNKNOWN_COM_ERROR, "Unsupported command");            break;        case ServerParse.USE:            UseHandler.handle(sql, c, rs >>> 8);            break;        case ServerParse.COMMIT:            c.commit();            break;        case ServerParse.ROLLBACK:            c.rollback();            break;        default:            c.execute(sql, rs);        }    }    我们再以default为例，看看接下来怎么进行。可以看到从这里已经开始涉及路由选择。    public void execute(String sql, int type) {        if (txInterrupted) {            writeErrMessage(ErrorCode.ER_YES, "Transaction error, need to rollback.");            return;        }        String db = this.schema;        if (db == null) {            writeErrMessage(ErrorCode.ER_NO_DB_ERROR, "No database selected");            return;        }        SchemaConfig schema = CobarServer.getInstance().getConfig().getSchemas().get(db);        if (schema == null) {            writeErrMessage(ErrorCode.ER_BAD_DB_ERROR, "Unknown database '" + db + "'");            return;        }        RouteResultset rrs = null;        try {            rrs = ServerRouter.route(schema, sql, this.charset, this);        } catch (SQLNonTransientException e) {            StringBuilder s = new StringBuilder();            LOGGER.warn(s.append(this).append(sql).toString(), e);            String msg = e.getMessage();            writeErrMessage(ErrorCode.ER_PARSE_ERROR, msg == null ? e.getClass().getSimpleName() : msg);            return;        }        session.execute(rrs, type);    }    对路由选择的每个节点执行sql操作    public void execute(RouteResultset rrs, int type) {        if (LOGGER.isDebugEnabled()) {            StringBuilder s = new StringBuilder();            LOGGER.debug(s.append(source).append(rrs).toString());        }        RouteResultsetNode[] nodes = rrs.getNodes();        if (nodes == null || nodes.length == 0) {            source.writeErrMessage(ErrorCode.ER_NO_DB_ERROR, "No dataNode selected");            return;        }        if (nodes.length == 1) {            singleNodeExecutor.execute(nodes[0], this, rrs.getFlag());        } else {            boolean autocommit = source.isAutocommit();            if (autocommit && isModifySQL(type)) {                autocommit = false;            }            multiNodeExecutor.execute(nodes, autocommit, this, rrs.getFlag());        }    }    执行计算并等待返回    BinaryPacket bin = ((MySQLChannel) c).execute(rrn, sc, autocommit);    接下来我们看看后端是怎么执行的。

后向连接
后向连接有两种方式，一种bio，一种nio；分别对应着BlockingSession与NonBlockingSession；在代码中，我下载的是版本默认使用BlockingSession，因此我们先分析BlockingSession，如下

        封装成执行数据包        CommandPacket packet = new CommandPacket();        packet.packetId = 0;        packet.command = MySQLPacket.COM_QUERY;        packet.arg = rrn.getStatement().getBytes(charset);        lastActiveTime = TimeUtil.currentTimeMillis();        packet.write(out);        out.flush();        这里是阻塞等待        BinaryPacket bin = receive();        long now = TimeUtil.currentTimeMillis();        if (now > lastActiveTime) {            recordSql(sc.getHost(), sc.getSchema(), rrn.getStatement());        }        lastActiveTime = now;        return bin;        直接使用的socket连接mysql后端        socket = new Socket();        socket.setTcpNoDelay(true);        socket.setTrafficClass(0x04 | 0x10);        socket.setPerformancePreferences(0, 2, 1);        socket.setReceiveBufferSize(RECV_BUFFER_SIZE);        socket.setSendBufferSize(SEND_BUFFER_SIZE);        socket.connect(new InetSocketAddress(dsc.getHost(), dsc.getPort()), SOCKET_CONNECT_TIMEOUT);

那么NonBlockingSession是怎样操作的呢，我们看看它的内部实现

        RouteResultsetNode[] nodes = rrs.getNodes();        if (nodes == null || nodes.length == 0) {            source.writeErrMessage(ErrorCode.ER_NO_DB_ERROR, "No dataNode selected");            return;        }        if (nodes.length == 1) {            singleNodeHandler = new SingleNodeHandler(nodes[0], this);            // singleNodeHandler.execute();        } else {            boolean autocommit = source.isAutocommit();            if (autocommit && isModifySQL(type)) {                autocommit = false;            }            multiNodeHandler = new MultiNodeQueryHandler(nodes, autocommit, this);            // multiNodeHandler.execute();        }        MultiNodeQueryHandler中的处理逻辑        ThreadPoolExecutor executor = session.getSource().getProcessor().getExecutor();        for (final RouteResultsetNode node : route) {            final MySQLConnection conn = session.getTarget(node);            if (conn != null) {                conn.setAttachment(node);                executor.execute(new Runnable() {                    @Override                    public void run() {                        _execute(conn, node);                    }                });            } else {                CobarConfig conf = CobarServer.getInstance().getConfig();                MySQLDataNode dn = conf.getDataNodes().get(node.getName());                dn.getConnection(this, node);            }        }        这里出现了MySQLConnection，前面提到的nio中的后端连接子类        private void _execute(MySQLConnection conn, RouteResultsetNode node) {            conn.setResponseHandler(this);            if (session.closed()) {                backendConnError(conn, "failed or cancelled by other thread");                return;            }            try {                conn.execute(node, session.getSource(), autocommit);            } catch (IOException e) {                connectionError(e, conn);            }        }        public void execute(RouteResultsetNode rrn, ServerConnection sc, boolean autocommit)                throws UnsupportedEncodingException {            StatusSync sync = new StatusSync(this, rrn, sc, autocommit);            statusSync = sync;            if (sync.isSync() || !sync.sync()) {                sync.execute();            }        }        public void execute() throws UnsupportedEncodingException {            executed = true;            CommandPacket packet = new CommandPacket();            packet.packetId = 0;            packet.command = MySQLPacket.COM_QUERY;            packet.arg = rrn.getStatement().getBytes(conn.getCharset());            conn.lastTime = TimeUtil.currentTimeMillis();            packet.write(conn);        }

未完，待续。。。