ServerBootstrap绑定和连接过程源码分析

转载自：http://blog.csdn.net/iter_zc/article/details/39349177
              https://segmentfault.com/a/1190000007283053
              https://segmentfault.com/a/1190000007403873

       典型的Netty服务器端代码如下，首先需要提供两个EventLoopGroup线程池，bossGroup是用来接受客户端TCP连接，workGroup是用来处理IO事件。然后指定采用何种Channel，Netty抽象了一组Channel的结构，和Java本身的Channel概念基本一致，NioServerSocketChannel相当于ServerSocketChannel。最后指定一组ChannelHandler来处理业务功能。（基于4.1.11.Final）

        EventLoopGroup bossGroup = new NioEventLoopGroup();          EventLoopGroup workerGroup = new NioEventLoopGroup();          try {              ServerBootstrap b = new ServerBootstrap();              b.group(bossGroup, workerGroup)                      .channel(NioServerSocketChannel.class)                    .option(ChannelOption.SO_BACKLOG, 100) // 设置tcp协议的请求等待队列                    .option(ChannelOption.SO_BACKLOG, 1024)                      .childHandler(new ChildChannelHandler());                            ChannelFuture f = b.bind(port).sync();              System.out.println("Netty time Server started at port " + port);              f.channel().closeFuture().sync();          } finally {              bossGroup.shutdownGracefully();              workerGroup.shutdownGracefully();          }  

       NioEventLoopGroup之前已经介绍过了，直接看ServerBootstrap.bind(port)，实际上是调用的AbstractBootstrap#bind()：

    /**     * Create a new {@link Channel} and bind it.     */    public ChannelFuture bind(int inetPort) {        return bind(new InetSocketAddress(inetPort));    }

    /**     * Create a new {@link Channel} and bind it.     */    public ChannelFuture bind(SocketAddress localAddress) {        validate();        if (localAddress == null) {            throw new NullPointerException("localAddress");        }        return doBind(localAddress);    }

    private ChannelFuture doBind(final SocketAddress localAddress) {        final ChannelFuture regFuture = initAndRegister();        final Channel channel = regFuture.channel();        if (regFuture.cause() != null) {            return regFuture;        }        if (regFuture.isDone()) {            // At this point we know that the registration was complete and successful.            ChannelPromise promise = channel.newPromise();            doBind0(regFuture, channel, localAddress, promise);            return promise;        } else {            // Registration future is almost always fulfilled already, but just in case it's not.            final PendingRegistrationPromise promise = new PendingRegistrationPromise(channel);            regFuture.addListener(new ChannelFutureListener() {                @Override                public void operationComplete(ChannelFuture future) throws Exception {                    Throwable cause = future.cause();                    if (cause != null) {                        promise.setFailure(cause);                    } else {                        promise.registered();                        doBind0(regFuture, channel, localAddress, promise);                    }                }            });            return promise;        }    }

    final ChannelFuture initAndRegister() {        Channel channel = null;        try {            channel = channelFactory.newChannel();            init(channel);        } catch (Throwable t) {            if (channel != null) {                // channel can be null if newChannel crashed (eg SocketException("too many open files"))                channel.unsafe().closeForcibly();            }            // as the Channel is not registered yet we need to force the usage of the GlobalEventExecutor            return new DefaultChannelPromise(channel, GlobalEventExecutor.INSTANCE).setFailure(t);        }        ChannelFuture regFuture = config().group().register(channel);        if (regFuture.cause() != null) {            if (channel.isRegistered()) {                channel.close();            } else {                channel.unsafe().closeForcibly();            }        }        return regFuture;    }

       initAndRegister代码着重分析一下，这里的channelFactory就是根据channel(NioServerSocketChannel.class)传入的一个Class对象，实例化ReflectiveChannelFactory对象，channelFactory返回的是NioServerSocketChannel:

    public NioServerSocketChannel() {        this(newSocket(DEFAULT_SELECTOR_PROVIDER));    }    public NioServerSocketChannel(ServerSocketChannel channel) {        super(null, channel, SelectionKey.OP_ACCEPT);        config = new NioServerSocketChannelConfig(this, javaChannel().socket());    }

       首先调用无参构造函数，该构造函数会调用下面那个构造函数，可以看到有个SelectionKey.OP_ACCEPT参数。NioServerSocketChannel继承关系链是：NioServerSocketChannel <- AbstractNioMessageChannel <- AbstractNioChannel <- AbstractChannel，右边是它的父类。
       逐层跟踪源码：

    protected AbstractNioMessageChannel(Channel parent, SelectableChannel ch, int readInterestOp) {        super(parent, ch, readInterestOp);    }

    protected AbstractNioChannel(Channel parent, SelectableChannel ch, int readInterestOp) {        super(parent);        this.ch = ch;        this.readInterestOp = readInterestOp;        try {            ch.configureBlocking(false);        } catch (IOException e) {            …        }    }

    protected AbstractChannel(Channel parent) {        this.parent = parent;        id = newId();        unsafe = newUnsafe();        pipeline = newChannelPipeline();    }

       这里，在AbstractNioChannel中，将readInterestOp设置为了SelectionKey.OP_ACCEPT，同时channel设置为非阻塞模式，在AbstractChannel中实例化一个unsafe和pipeline，具体的实例分别是NioMessageUnsafe和DefaultChannelPipeline。

       回到AbstractBootstrap#initAndRegister方法中，newChannel()完成之后，就对该channel做init工作，实质上调用的是ServerBootstrap中的init：

    @Override    void init(Channel channel) throws Exception {        final Map<ChannelOption<?>, Object> options = options0();        synchronized (options) {            setChannelOptions(channel, options, logger);        }        final Map<AttributeKey<?>, Object> attrs = attrs0();        synchronized (attrs) {            for (Entry<AttributeKey<?>, Object> e: attrs.entrySet()) {                @SuppressWarnings("unchecked")                AttributeKey<Object> key = (AttributeKey<Object>) e.getKey();                channel.attr(key).set(e.getValue());            }        }        ChannelPipeline p = channel.pipeline();        final EventLoopGroup currentChildGroup = childGroup;        final ChannelHandler currentChildHandler = childHandler;        final Entry<ChannelOption<?>, Object>[] currentChildOptions;        final Entry<AttributeKey<?>, Object>[] currentChildAttrs;        synchronized (childOptions) {            currentChildOptions = childOptions.entrySet().toArray(newOptionArray(childOptions.size()));        }        synchronized (childAttrs) {            currentChildAttrs = childAttrs.entrySet().toArray(newAttrArray(childAttrs.size()));        }        p.addLast(new ChannelInitializer<Channel>() {            @Override            public void initChannel(final Channel ch) throws Exception {                final ChannelPipeline pipeline = ch.pipeline();                ChannelHandler handler = config.handler();                if (handler != null) {                    pipeline.addLast(handler);                }                ch.eventLoop().execute(new Runnable() {                    @Override                    public void run() {                        pipeline.addLast(new ServerBootstrapAcceptor(                                ch, currentChildGroup, currentChildHandler, currentChildOptions, currentChildAttrs));                    }                });            }        });    }

       先给NioServerSocketChannel设置Socket参数和附加属性，然后给NioServerSocketChannel关联的DefaultChannelPipeline中添加了一个ChannelInitializer,而这个 ChannelInitializer中添加了一个关键的ServerBootstrapAcceptor handler，这里会把currentChildGroup即workerGroup作为构造函数参数传入ServerBootstrapAcceptor，还有childHandler、childOptions、childAttrs，childHandler即服务器端代码中bootstrap.childHandler(…)所指的内容。

       再回到AbstractBootstrap#initAndRegister方法继续向下看，config().group().register(channel)会将NioServerSocketChannel注册到EventLoop中。config().group()获取的EventLoopGroup是AbstractBootstrap的成员，该值即ServerBootstrap.group(parentGroup, childGroup)调用时的parentGroup，即常说到的bossGroup。
       因此group().register()调用的实际上是MultithreadEventLoopGroup.register()方法：

    @Override    public ChannelFuture register(Channel channel) {        return next().register(channel);    }

       next()返回EventExecutor[] children中的一个EventExecutor来完成register。 EventExecutor实际上就是NioEventLoop，即从bossGroup中选一个NioEventLoop出来，最终调用NioEventLoop的父类SingleThreadEventLoop的register方法。

    @Override    public ChannelFuture register(Channel channel) {        return register(new DefaultChannelPromise(channel, this));    }    @Override    public ChannelFuture register(final ChannelPromise promise) {        ObjectUtil.checkNotNull(promise, "promise");        promise.channel().unsafe().register(this, promise);        return promise;    }

       在channel().unsafe().register()中会将EventLoop赋值给AbstractChannel内部的 eventLoop字段，到这里就完成了bossGroup 的EventLoop与NioServerSocketChannel的关联过程：

        @Override        public final void register(EventLoop eventLoop, final ChannelPromise promise) {            ……            AbstractChannel.this.eventLoop = eventLoop;            if (eventLoop.inEventLoop()) {                register0(promise);            } else {                try {                    eventLoop.execute(new Runnable() {                        @Override                        public void run() {                            register0(promise);                        }                    });                } catch (Throwable t) {                    logger.warn(                            "Force-closing a channel whose registration task was not accepted by an event loop: {}",                            AbstractChannel.this, t);                    closeForcibly();                    closeFuture.setClosed();                    safeSetFailure(promise, t);                }            }        }

       在注册NioServerSocketChannel的过程中,会在AbstractChannel#AbstractUnsafe.register中调用 eventLoop.execute方法, 由于整个代码都是在主线程中运行的, 因此上面的 eventLoop.inEventLoop() 就为false, 于是进入到else分支, 在这个分支中调用了eventLoop.execute。再次强调一下，这里的eventLoop是bossGroup中的一个eventLoop。eventLoop是一个NioEventLoop的实例,最终调用的是 SingleThreadEventExecutor.execute:

    @Override    public void execute(Runnable task) {        if (task == null) {            throw new NullPointerException("task");        }        boolean inEventLoop = inEventLoop();        if (inEventLoop) {            addTask(task);        } else {            startThread();            addTask(task);            if (isShutdown() && removeTask(task)) {                reject();            }        }        if (!addTaskWakesUp && wakesUpForTask(task)) {            wakeup(inEventLoop);        }    }

       已经分析过inEventLoop == false, 因此执行到else分支, 在这里就调用了startThread() 方法来启动 SingleThreadEventExecutor内部关联的Java线程。

    private void startThread() {        if (state == ST_NOT_STARTED) {            if (STATE_UPDATER.compareAndSet(this, ST_NOT_STARTED, ST_STARTED)) {                doStartThread();            }        }    }

       STATE_UPDATER是SingleThreadEventExecutor内部维护的一个属性, 它的作用是标识当前的thread的状态. 在初始的时候, STATE_UPDATER == ST_NOT_STARTED, 因此第一次调用 startThread()方法时, 就会进入到if语句内, 进而调用到 doStartThread()。
       startThread()方法之后接着是addTask(task)方法，addTask是将task，即：

           new Runnable() {                @Override                public void run() {                    register0(promise);                }           }

       放入到taskQueue队列中去，之后会有线程从taskQueue取出任务执行。register0的主要作用是注册ChannelPipeLine。

       来看看AbstractChannel#register0(promise)做了什么：

        private void register0(ChannelPromise promise) {            try {                ……                boolean firstRegistration = neverRegistered;                doRegister();                neverRegistered = false;                registered = true;                ……                pipeline.invokeHandlerAddedIfNeeded();                safeSetSuccess(promise);                pipeline.fireChannelRegistered();                // Only fire a channelActive if the channel has never been registered. This prevents firing                // multiple channel actives if the channel is deregistered and re-registered.                if (isActive()) {                    if (firstRegistration) {                        pipeline.fireChannelActive();                    } else if (config().isAutoRead()) {                        // This channel was registered before and autoRead() is set. This means we need to begin read                        // again so that we process inbound data.                        beginRead();                    }                }            } catch (Throwable t) {                ……            }        }

       doRegister()实际上是一个空方法，然后将promise设置为success状态，表示注册过程完成。pipeline.fireChannelRegistered()触发一个ChannelRegistered事件，会从DefaultChannelPipeline的head开始，最终传递tail即TailContext来处理该事件，TailContext的channelRegistered方法也是个空实现。 

       回头来看AbstractBootstrap#doBind，判断initAndRegister返回的regFuture是否完成，在AbstractChannel#register0(promise)中，有设置success状态，因此会继续执行下面的doBind0方法：

    private static void doBind0(            final ChannelFuture regFuture, final Channel channel,            final SocketAddress localAddress, final ChannelPromise promise) {        // This method is invoked before channelRegistered() is triggered.  Give user handlers a chance to set up        // the pipeline in its channelRegistered() implementation.        channel.eventLoop().execute(new Runnable() {            @Override            public void run() {                if (regFuture.isSuccess()) {                    channel.bind(localAddress, promise).addListener(ChannelFutureListener.CLOSE_ON_FAILURE);                } else {                    promise.setFailure(regFuture.cause());                }            }        });    }

       这里的channel即NioServerSocketChannel，其关联的eventLoop即bossGroup中的一个，eventLoop的execute方法，之前有说过eventLoop的execute方法，会将task放在taskQueue中等待线程来执行。
       那先来看channel.bind，即DefaultChannelPipeline#bind:

    public final ChannelFuture bind(SocketAddress localAddress, ChannelPromise promise) {        return tail.bind(localAddress, promise);    }

    @Override    public ChannelFuture bind(final SocketAddress localAddress, final ChannelPromise promise) {        ……        final AbstractChannelHandlerContext next = findContextOutbound();        EventExecutor executor = next.executor();        if (executor.inEventLoop()) {            next.invokeBind(localAddress, promise);        } else {            safeExecute(executor, new Runnable() {                @Override                public void run() {                    next.invokeBind(localAddress, promise);                }            }, promise, null);        }        return promise;    }

    private AbstractChannelHandlerContext findContextOutbound() {        AbstractChannelHandlerContext ctx = this;        do {            ctx = ctx.prev;        } while (!ctx.outbound);        return ctx;    }

    private void invokeBind(SocketAddress localAddress, ChannelPromise promise) {        if (invokeHandler()) {            try {                ((ChannelOutboundHandler) handler()).bind(this, localAddress, promise);            } catch (Throwable t) {                notifyOutboundHandlerException(t, promise);            }        } else {            bind(localAddress, promise);        }    }

       DefaultChannelPipeline中head即HeadContext是最开始的AbstractChannelHandlerContext，因此调用它的bind方法：

        public void bind(                ChannelHandlerContext ctx, SocketAddress localAddress, ChannelPromise promise)                throws Exception {            unsafe.bind(localAddress, promise);        }

        @Override        public final void bind(final SocketAddress localAddress, final ChannelPromise promise) {            …            boolean wasActive = isActive();            try {                doBind(localAddress);            } catch (Throwable t) {                safeSetFailure(promise, t);                closeIfClosed();                return;            }            if (!wasActive && isActive()) {                invokeLater(new Runnable() {                    @Override                    public void run() {                        pipeline.fireChannelActive();                    }                });            }            safeSetSuccess(promise);        }

       doBind(localAddress)最终调用的是NioServerSocketChannel#bind来完成端口的绑定：

    @Override    protected void doBind(SocketAddress localAddress) throws Exception {        if (PlatformDependent.javaVersion() >= 7) {            javaChannel().bind(localAddress, config.getBacklog());        } else {            javaChannel().socket().bind(localAddress, config.getBacklog());        }    }

       然后会调用pipeline.fireChannelActive()触发ChannelActive事件：

    public final ChannelPipeline fireChannelActive() {        AbstractChannelHandlerContext.invokeChannelActive(head);        return this;    }

       会调用head即HeadContext的channelActive：

        public void channelActive(ChannelHandlerContext ctx) throws Exception {            ctx.fireChannelActive();            readIfIsAutoRead();        }

        private void readIfIsAutoRead() {            if (channel.config().isAutoRead()) {                channel.read();            }        }

       HeadContext会主动发起一个outbound的读事件，通过Pipeline最后传递到HeadContext的read方法。HeadHandler处理read事件时，实际上调用了unsafe的beginRead方法，最后到了AbstractNioChannel的doBeginRead()：

    @Override    protected void doBeginRead() throws Exception {        // Channel.read() or ChannelHandlerContext.read() was called        final SelectionKey selectionKey = this.selectionKey;        if (!selectionKey.isValid()) {            return;        }        readPending = true;        final int interestOps = selectionKey.interestOps();        if ((interestOps & readInterestOp) == 0) {            selectionKey.interestOps(interestOps | readInterestOp);        }    }

       根据之前的介绍已经知道，此时的readInterestOps是OP_ACCEPT = 16， selectionKey.interestOps=0；因此进入到 selectionKey.interestOps(interestOps|readInterestOp); 使用或操作设置selectionKey的interestOps为OP_ACCEPT，表示开始监听网络连接。

       前面提到了SingleThreadEventExecutor#startThread() 方法中会调用doStartThread()来启动SingleThreadEventExecutor内部关联的Java线程：

    private void doStartThread() {        assert thread == null;        executor.execute(new Runnable() {            @Override            public void run() {                thread = Thread.currentThread();                if (interrupted) {                    thread.interrupt();                }                boolean success = false;                updateLastExecutionTime();                try {                    SingleThreadEventExecutor.this.run();                    success = true;                } catch (Throwable t) {                    logger.warn("Unexpected exception from an event executor: ", t);                } finally {                    ……                }            }        });    }

       这里面最重要的是SingleThreadEventExecutor.this.run()，实际上是NioEventLoop#run:

    @Override    protected void run() {        for (;;) {            try {                ……                cancelledKeys = 0;                needsToSelectAgain = false;                final int ioRatio = this.ioRatio;                if (ioRatio == 100) {                    try {                        processSelectedKeys();                    } finally {                        // Ensure we always run tasks.                        runAllTasks();                    }                } else {                    final long ioStartTime = System.nanoTime();                    try {                        processSelectedKeys();                    } finally {                        // Ensure we always run tasks.                        final long ioTime = System.nanoTime() - ioStartTime;                        runAllTasks(ioTime * (100 - ioRatio) / ioRatio);                    }                }            } catch (Throwable t) {                handleLoopException(t);            }            ……        }    }

       这里会调用processSelectedKeys()来处理感兴趣的OP_ACCEPT事件，调用链是NioEventLoop#processSelectedKey --- NioMessageUnsafe#read，在NioMessageUnsafe中会触发pipeline的事件；同时，还会有runAllTasks()，来处理taskQueue队列中的任务。可以设置ioRatio，来控制执行IO事件和非IO事件的时间比。

       看下NioMessageUnsafe#read：

        @Override        public void read() {            ……            final ChannelConfig config = config();            final ChannelPipeline pipeline = pipeline();            final RecvByteBufAllocator.Handle allocHandle = unsafe().recvBufAllocHandle();            allocHandle.reset(config);            boolean closed = false;            Throwable exception = null;            try {                try {                    do {                        int localRead = doReadMessages(readBuf);                        …                    } while (allocHandle.continueReading());                } catch (Throwable t) {                    exception = t;                }                int size = readBuf.size();                for (int i = 0; i < size; i ++) {                    readPending = false;                    pipeline.fireChannelRead(readBuf.get(i));                }                readBuf.clear();                allocHandle.readComplete();                pipeline.fireChannelReadComplete();                if (exception != null) {                    closed = closeOnReadError(exception);                    pipeline.fireExceptionCaught(exception);                }                if (closed) {                    inputShutdown = true;                    if (isOpen()) {                        close(voidPromise());                    }                }            } finally {                // Check if there is a readPending which was not processed yet.                // This could be for two reasons:                // * The user called Channel.read() or ChannelHandlerContext.read() in channelRead(...) method                // * The user called Channel.read() or ChannelHandlerContext.read() in channelReadComplete(...) method                //                // See https://github.com/netty/netty/issues/2254                if (!readPending && !config.isAutoRead()) {                    removeReadOp();                }            }        }

       doReadMessages方法实际上接受客户端连接，并创建了NioSocketChannel：

    protected int doReadMessages(List<Object> buf) throws Exception {        SocketChannel ch = SocketUtils.accept(javaChannel());        try {            if (ch != null) {                buf.add(new NioSocketChannel(this, ch));                return 1;            }        } catch (Throwable t) {            ……        }        return 0;    }

    public NioSocketChannel(Channel parent, SocketChannel socket) {        super(parent, socket);        config = new NioSocketChannelConfig(this, socket.socket());    }

    protected AbstractNioByteChannel(Channel parent, SelectableChannel ch) {        super(parent, ch, SelectionKey.OP_READ);    }

       同之前的NioServerSocketChannel的构建类似，构造NioSocketChannel时传入了SelectionKey.OP_READ。

       之后对readBuf中的每个NioSocketChannel触发pipeline的ChannelRead事件：

    public final ChannelPipeline fireChannelRead(Object msg) {        AbstractChannelHandlerContext.invokeChannelRead(head, msg);        return this;    }

       ChannelRead会从head开始传递，调用pipeline链中的InboundContext的channelRead方法，之前ServerBootstrapAcceptor是加到了pipeline链中的，而且它继承了ChannelInboundHandlerAdapter，因此会调用ServerBootstrapAcceptor的channelRead方法：

        public void channelRead(ChannelHandlerContext ctx, Object msg) {            final Channel child = (Channel) msg;            child.pipeline().addLast(childHandler);            setChannelOptions(child, childOptions, logger);            for (Entry<AttributeKey<?>, Object> e: childAttrs) {                child.attr((AttributeKey<Object>) e.getKey()).set(e.getValue());            }            try {                childGroup.register(child).addListener(new ChannelFutureListener() {                    @Override                    public void operationComplete(ChannelFuture future) throws Exception {                        if (!future.isSuccess()) {                            forceClose(child, future.cause());                        }                    }                });            } catch (Throwable t) {                forceClose(child, t);            }        }

       这里的msg 即channel，是一个NioSocketChannel的实例, childGroup即workerGroup，于是workerGroup的EventLoop和NioSocketChannel 关联了。注意这里的childGroup.register(child)，类似与前面的介绍，最终调用了AbstractNioChannel#doBeginRead方法，将SelectionKey.OP_READ添加到了selectionKey中。