Netty 作为服务器端源代码剖析

1 -ServerBootstrap绑定本地端口。

AbstractNioMessageChannel$NioMessageUnsafe(AbstractChannel$AbstractUnsafe).register(EventLoop, ChannelPromise)

提交register0任务时,是启动NioEventLoop线程的地方。

NioEventLoop(SingleThreadEventExecutor).doStartThread() line: 730NioEventLoop(SingleThreadEventExecutor).startThread() line: 724NioEventLoop(SingleThreadEventExecutor).execute(Runnable) line: 671AbstractNioMessageChannel$NioMessageUnsafe(AbstractChannel$AbstractUnsafe).register(EventLoop, ChannelPromise) line: 468NioEventLoop(SingleThreadEventLoop).register(ChannelPromise) line: 56NioEventLoop(SingleThreadEventLoop).register(Channel) line: 50NioEventLoopGroup(MultithreadEventLoopGroup).register(Channel) line: 75ServerBootstrap(AbstractBootstrap<B,C>).initAndRegister() line: 327ServerBootstrap(AbstractBootstrap<B,C>).doBind(SocketAddress) line: 282ServerBootstrap(AbstractBootstrap<B,C>).bind(SocketAddress) line: 278

在SingleThreadEventExecutor 类中，doStartThread方法启动一个任务，该任务执行SingleThreadEventExecutor.this.run()；方法。

public final class NioEventLoop extends SingleThreadEventLoop {   for (;;) {              /**     * Poll all tasks from the task queue and run them via {@link Runnable#run()} method.  This method stops running     * the tasks in the task queue and returns if it ran longer than {@code timeoutNanos}.     */    protected boolean runAllTasks(long timeoutNanos) {             }    }}

NioEventLoop在轮询runAllTasks()任务。NioEventLoop(SingleThreadEventExecutor).runAllTasks(long)

AbstractBootstrap<B,C>.doBind0(ChannelFuture, Channel, SocketAddress, ChannelPromise)

    private static void doBind0(            final ChannelFuture regFuture, final Channel channel,            final SocketAddress localAddress, final ChannelPromise promise) {        // This method is invoked before channelRegistered() is triggered.  Give user handlers a chance to set up        // the pipeline in its channelRegistered() implementation.        channel.eventLoop().execute(new OneTimeTask() {            @Override            public void run() {                if (regFuture.isSuccess()) {                    channel.bind(localAddress, promise).addListener(ChannelFutureListener.CLOSE_ON_FAILURE);                } else {                    promise.setFailure(regFuture.cause());                }            }        });    }

在提交OneTimeTask任务时，判断是否与EventLoop想管理了。NioEventLoop(SingleThreadEventExecutor).execute(Runnable)

    @Override    public void execute(Runnable task) {        if (task == null) {            throw new NullPointerException("task");        }        boolean inEventLoop = inEventLoop();        if (inEventLoop) {            addTask(task);        } else {            startThread();            addTask(task);            if (isShutdown() && removeTask(task)) {                reject();            }        }        if (!addTaskWakesUp && wakesUpForTask(task)) {            wakeup(inEventLoop);        }    }

把任务放置在private final Queue<Runnable> taskQueue;队列中。在NioEventLoop 中会执行该任务。

NioEventLoop线程是在NioEventLoop(SingleThreadEventExecutor).execute(Runnable) 方法启动的。

NioEventLoop.run()  方法中存在for (;;) {} 循环来执行所有任务。

2 -NioEventLoop来处理IO事件。NioEventLoop.processSelectedKey(SelectionKey, AbstractNioChannel) 处理IO读写。

3- 当NioEventLoop处理accept事件时，ServerBootstrap$ServerBootstrapAcceptor.channelRead(ChannelHandlerContext, Object) 来处理。

    private static class ServerBootstrapAcceptor extends ChannelInboundHandlerAdapter {        private final EventLoopGroup childGroup;        private final ChannelHandler childHandler;        private final Entry<ChannelOption<?>, Object>[] childOptions;        private final Entry<AttributeKey<?>, Object>[] childAttrs;        ServerBootstrapAcceptor(                EventLoopGroup childGroup, ChannelHandler childHandler,                Entry<ChannelOption<?>, Object>[] childOptions, Entry<AttributeKey<?>, Object>[] childAttrs) {            this.childGroup = childGroup;            this.childHandler = childHandler;            this.childOptions = childOptions;            this.childAttrs = childAttrs;        }        @Override        @SuppressWarnings("unchecked")        public void channelRead(ChannelHandlerContext ctx, Object msg) {            final Channel child = (Channel) msg;            child.pipeline().addLast(childHandler);            for (Entry<ChannelOption<?>, Object> e: childOptions) {                try {                    if (!child.config().setOption((ChannelOption<Object>) e.getKey(), e.getValue())) {                        logger.warn("Unknown channel option: " + e);                    }                } catch (Throwable t) {                    logger.warn("Failed to set a channel option: " + child, t);                }            }            for (Entry<AttributeKey<?>, Object> e: childAttrs) {                child.attr((AttributeKey<Object>) e.getKey()).set(e.getValue());            }            try {                childGroup.register(child).addListener(new ChannelFutureListener() {                    @Override                    public void operationComplete(ChannelFuture future) throws Exception {                        if (!future.isSuccess()) {                            forceClose(child, future.cause());                        }                    }                });            } catch (Throwable t) {                forceClose(child, t);            }        }        private static void forceClose(Channel child, Throwable t) {            child.unsafe().closeForcibly();            logger.warn("Failed to register an accepted channel: " + child, t);        }        @Override        public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {            final ChannelConfig config = ctx.channel().config();            if (config.isAutoRead()) {                // stop accept new connections for 1 second to allow the channel to recover                // See https://github.com/netty/netty/issues/1328                config.setAutoRead(false);                ctx.channel().eventLoop().schedule(new OneTimeTask() {                    @Override                    public void run() {                        config.setAutoRead(true);                    }                }, 1, TimeUnit.SECONDS);            }            // still let the exceptionCaught event flow through the pipeline to give the user            // a chance to do something with it            ctx.fireExceptionCaught(cause);        }    }

在ServerBootstarpAcceptor中，会把SocketChannel交给childGroup来处理。

4 -注意：

    static void invokeChannelRead(final AbstractChannelHandlerContext next, Object msg) {        final Object m = next.pipeline.touch(ObjectUtil.checkNotNull(msg, "msg"), next);        EventExecutor executor = next.executor();        if (executor.inEventLoop()) {            next.invokeChannelRead(m);        } else {            executor.execute(new OneTimeTask() {                @Override                public void run() {                    next.invokeChannelRead(m);                }            });        }    }

在所有处理业务逻辑时，都是先判断是否由当前的EventLoop相关联在一起处理的。如果是当前EventLoop线程来处理的，就继续叫给当前线程来处理，减少线程的切换和共享数据的访问。当业务逻辑不是当前线程来处理时，在创建一个OneTimeTask任务，并把任务放置到任务队列中，然后由线程池来处理。

所以所有的业务都是有当前线程来处理的。

在NioEventLoop extends  SingleThreadEventExecutor 类中，所有的OneTimeTask都被加入到private final Queue<Runnable> taskQueue; 队列中，然后迭代这个队列。

在NioEventLoop中会同步处理NioEventLoop.processSelectedKey(SelectionKey, AbstractNioChannel) 方法。

    private void processSelectedKey(SelectionKey k, AbstractNioChannel ch) {        final AbstractNioChannel.NioUnsafe unsafe = ch.unsafe();        if (!k.isValid()) {            final EventLoop eventLoop;            try {                eventLoop = ch.eventLoop();            } catch (Throwable ignored) {                // If the channel implementation throws an exception because there is no event loop, we ignore this                // because we are only trying to determine if ch is registered to this event loop and thus has authority                // to close ch.                return;            }            // Only close ch if ch is still registerd to this EventLoop. ch could have deregistered from the event loop            // and thus the SelectionKey could be cancelled as part of the deregistration process, but the channel is            // still healthy and should not be closed.            // See https://github.com/netty/netty/issues/5125            if (eventLoop != this || eventLoop == null) {                return;            }            // close the channel if the key is not valid anymore            unsafe.close(unsafe.voidPromise());            return;        }        try {            int readyOps = k.readyOps();            // Also check for readOps of 0 to workaround possible JDK bug which may otherwise lead            // to a spin loop            if ((readyOps & (SelectionKey.OP_READ | SelectionKey.OP_ACCEPT)) != 0 || readyOps == 0) {                unsafe.read();                if (!ch.isOpen()) {                    // Connection already closed - no need to handle write.                    return;                }            }            if ((readyOps & SelectionKey.OP_WRITE) != 0) {                // Call forceFlush which will also take care of clear the OP_WRITE once there is nothing left to write                ch.unsafe().forceFlush();            }            if ((readyOps & SelectionKey.OP_CONNECT) != 0) {                // remove OP_CONNECT as otherwise Selector.select(..) will always return without blocking                // See https://github.com/netty/netty/issues/924                int ops = k.interestOps();                ops &= ~SelectionKey.OP_CONNECT;                k.interestOps(ops);                unsafe.finishConnect();            }        } catch (CancelledKeyException ignored) {            unsafe.close(unsafe.voidPromise());        }    }

在processSelectedKeys会处理所有下面的事件。

    /**     * Operation-set bit for read operations.     *     * <p> Suppose that a selection key's interest set contains     * <tt>OP_READ</tt> at the start of a <a     * href="Selector.html#selop">selection operation</a>.  If the selector     * detects that the corresponding channel is ready for reading, has reached     * end-of-stream, has been remotely shut down for further reading, or has     * an error pending, then it will add <tt>OP_READ</tt> to the key's     * ready-operation set and add the key to its selected-key set.  </p>     */    public static final int OP_READ = 1 << 0;    /**     * Operation-set bit for write operations.     *     * <p> Suppose that a selection key's interest set contains     * <tt>OP_WRITE</tt> at the start of a <a     * href="Selector.html#selop">selection operation</a>.  If the selector     * detects that the corresponding channel is ready for writing, has been     * remotely shut down for further writing, or has an error pending, then it     * will add <tt>OP_WRITE</tt> to the key's ready set and add the key to its     * selected-key set.  </p>     */    public static final int OP_WRITE = 1 << 2;    /**     * Operation-set bit for socket-connect operations.     *     * <p> Suppose that a selection key's interest set contains     * <tt>OP_CONNECT</tt> at the start of a <a     * href="Selector.html#selop">selection operation</a>.  If the selector     * detects that the corresponding socket channel is ready to complete its     * connection sequence, or has an error pending, then it will add     * <tt>OP_CONNECT</tt> to the key's ready set and add the key to its     * selected-key set.  </p>     */    public static final int OP_CONNECT = 1 << 3;    /**     * Operation-set bit for socket-accept operations.     *     * <p> Suppose that a selection key's interest set contains     * <tt>OP_ACCEPT</tt> at the start of a <a     * href="Selector.html#selop">selection operation</a>.  If the selector     * detects that the corresponding server-socket channel is ready to accept     * another connection, or has an error pending, then it will add     * <tt>OP_ACCEPT</tt> to the key's ready set and add the key to its     * selected-key set.  </p>     */    public static final int OP_ACCEPT = 1 << 4;

其中读写事件由NioMessageUnsafe方法处理：

    private final class NioMessageUnsafe extends AbstractNioUnsafe {        private final List<Object> readBuf = new ArrayList<Object>();        @Override        public void read() {            assert eventLoop().inEventLoop();            final ChannelConfig config = config();            final ChannelPipeline pipeline = pipeline();            final RecvByteBufAllocator.Handle allocHandle = unsafe().recvBufAllocHandle();            allocHandle.reset(config);            boolean closed = false;            Throwable exception = null;            try {                try {                    do {                        int localRead = doReadMessages(readBuf);                        if (localRead == 0) {                            break;                        }                        if (localRead < 0) {                            closed = true;                            break;                        }                        allocHandle.incMessagesRead(localRead);                    } while (allocHandle.continueReading());                } catch (Throwable t) {                    exception = t;                }                int size = readBuf.size();                for (int i = 0; i < size; i ++) {                    readPending = false;                    pipeline.fireChannelRead(readBuf.get(i));                }                readBuf.clear();                allocHandle.readComplete();                pipeline.fireChannelReadComplete();                if (exception != null) {                    if (exception instanceof IOException && !(exception instanceof PortUnreachableException)) {                        // ServerChannel should not be closed even on IOException because it can often continue                        // accepting incoming connections. (e.g. too many open files)                        closed = !(AbstractNioMessageChannel.this instanceof ServerChannel);                    }                    pipeline.fireExceptionCaught(exception);                }                if (closed) {                    inputShutdown = true;                    if (isOpen()) {                        close(voidPromise());                    }                }            } finally {                // Check if there is a readPending which was not processed yet.                // This could be for two reasons:                // * The user called Channel.read() or ChannelHandlerContext.read() in channelRead(...) method                // * The user called Channel.read() or ChannelHandlerContext.read() in channelReadComplete(...) method                //                // See https://github.com/netty/netty/issues/2254                if (!readPending && !config.isAutoRead()) {                    removeReadOp();                }            }        }    }

所有的事件又由AbstractChannelHandlerContext 静态方法来调用。在AbstractChannelHandlerContext又是通过下面提交OneTimeTask把任务提交到线程中。让线程池来处理。

AbstractChannelHandlerContext{     static void invokeChannelRegistered(final AbstractChannelHandlerContext next) {        EventExecutor executor = next.executor();        if (executor.inEventLoop()) {            next.invokeChannelRegistered();        } else {            executor.execute(new OneTimeTask() {                @Override                public void run() {                    next.invokeChannelRegistered();                }            });        }    }}

在次看来，Mina和Netty的线程模型是一模一样滴。不过Netty是在mina的基础上继承和发展过来的，又经过众多大公司实践优化过，必定青出于蓝而胜于蓝。