Cassandra 源码解析 1：网络通信

org.apache.cassandra.thrift -  Server 端入口

CassandraDaemon: 启动类

CassandraServer implements org.apache.cassandra.thrift.Cassandra.Iface : 接口类，类似于web service中的skeleton

CustomTThreadPoolServer extends  org.apache.thrift.server.TServer ：监听类，监听thrift client请求

 

org.apache.cassandra.net

1. message 监听线程

启动 SocketThread 线程 "ACCEPT-" + localEp

StorageService.initServer

      -> MessagingService.instance.listen(FBUtilities.getLocalAddress());

         -> SocketThread.start()

监听7000端口，接受socket 连接. 这里有一点与一般SocketServer/Socket 编程不一样的是使用ServerSocketChannel来构造SocketServer，而不是通过 new ServerSocket(int port) 来构造.

每个接收的socket, 生成一个IncomingTcpConnectio线程

 

//MessagingService.listenServerSocketChannel serverChannel = ServerSocketChannel.open();final ServerSocket ss = serverChannel.socket();ss.setReuseAddress(true);//SocketThread public void run() { while (true) { try { Socket socket = server.accept(); new IncomingTcpConnection(socket).start(); } catch (AsynchronousCloseException e) { // this happens when another thread calls close(). logger_.info("MessagingService shutting down server thread."); break; } catch (IOException e) { throw new RuntimeException(e); } } }

 

 

2. message read 线程

每个 IncomingTcpConnection 会成为一个独立的线程，loop 阻塞式的读取消息

2.1 comm protocol

 每次读取的非stream消息格式如下(stream 消息另作分析)：

  | constant magic  | 4 byte |

  |header | 4 byte|

  | content size | 4 byte|

  | content | content size * 1 byte|

MessageService.packIt(byte[] content, boolean compress) 对应content打包过程

public IncomingTcpConnection(Socket socket) { this.socket = socket; try { input = new DataInputStream(socket.getInputStream()); } catch (IOException e) { throw new IOError(e); } }while (true) { try { MessagingService.validateMagic(input.readInt()); int header = input.readInt(); int type = MessagingService.getBits(header, 1, 2); boolean isStream = MessagingService.getBits(header, 3, 1) == 1; int version = MessagingService.getBits(header, 15, 8); if (isStream) { new IncomingStreamReader(socket.getChannel()).read(); } else { int size = input.readInt(); byte[] contentBytes = new byte[size]; input.readFully(contentBytes); MessagingService.getDeserializationExecutor().submit(new MessageDeserializationTask(new ByteArrayInputStream(contentBytes))); } } catch (EOFException e) { if (logger.isTraceEnabled()) logger.trace("eof reading from socket; closing", e); break; } catch (IOException e) { if (logger.isDebugEnabled()) logger.debug("error reading from socket; closing", e); break; } }

 

每个读到的消息content(byte[])，生成一个MessageDeserializationTask，交由单线程pool(MessageService.messageDeserializerExecutor_,)来运行 - DebuggableThreadPoolExecutor并且实现了 JMXEnabledThreadPoolExecutorMBean，另作分析

 

从第p位(从右到左(从低到高)数)开始，连续n位(低位方向)的值。运算优先级是加减，然后移位，然后与或

public static int getBits(int x, int p, int n){ return x >>> (p + 1) - n & ~(-1 << n);}

 

3. message 反序列化线程

MessageDeserializationTask 在 messageDeserializerExecutor 中运行，从content(byte[])反序列化生成Message对象

关于Message序列化与反序列化另作分析. 得到Message后，调用MessageService.receive(Message)

4. message handler 线程

4.1 Stage Thread poll

MessageService.receive 中生成 MessageDeliveryTask，每个Task会在不同Stage的ThreadPool中运行

StageManager 中对不同message type的ThreadPool初始化

 

MUTATION_STAGE : 32 // 写

READ_STAGE : 8 // 读

RESPONSE_STAGE : Max(2, process num) // 读写的response

STREAM_STAGE : 1 // ?

GOSSIP_STAGE : 1 // member detection

AE_SERVICE_STAGE : 1 // ?

LOADBALANCE_STAGE : 1 // ?

 

4.2 Message handler

MessageDeliveryTask 从MessagingService.instance中根据header中的verb找到对应的message handler(StorageService中注册)，调用   IVerbHandler.doVerb(message_);

4.2.1 RowMutationVerbHandler

byte[] bytes = message.getMessageBody(); ByteArrayInputStream buffer = new ByteArrayInputStream(bytes); try { RowMutation rm = RowMutation.serializer().deserialize(new DataInputStream(buffer)); if (logger_.isDebugEnabled()) logger_.debug("Applying " + rm); /* Check if there were any hints in this message */ byte[] hintedBytes = message.getHeader(RowMutation.HINT); if (hintedBytes != null) { assert hintedBytes.length > 0; ByteBuffer bb = ByteBuffer.wrap(hintedBytes); byte[] addressBytes = new byte[FBUtilities.getLocalAddress().getAddress().length]; while (bb.remaining() > 0) { bb.get(addressBytes); InetAddress hint = InetAddress.getByAddress(addressBytes); if (logger_.isDebugEnabled()) logger_.debug("Adding hint for " + hint); RowMutation hintedMutation = new RowMutation(Table.SYSTEM_TABLE, rm.getTable()); hintedMutation.addHints(rm.key(), addressBytes); hintedMutation.apply(); } } Table.open(rm.getTable()).apply(rm, bytes, true); WriteResponse response = new WriteResponse(rm.getTable(), rm.key(), true); Message responseMessage = WriteResponse.makeWriteResponseMessage(message, response); if (logger_.isDebugEnabled()) logger_.debug(rm + " applied. Sending response to " + message.getMessageId() + "@" + message.getFrom()); MessagingService.instance.sendOneWay(responseMessage, message.getFrom()); } catch (IOException e) { logger_.error("Error in row mutation", e); }

1. 反序列化Message body得到RowMutation
2. 判断是否是hinted写入(target server is down, 由该Server来保存记录) 如果是hinted写入，则作一个标记
3. 将RowMutation写入到commit log，等待提交，
4. 生成一个WriteResponse，返回一个Stage为RESPONSE_STAGE, verb为READ_RESPONSE的message，body包含table name，key，true(表示提交成功)，注意Message id 和Request中的Message一致
5. 调用 MessagingService.sendOneWay 将消息发送出去

4.2.2 ResponseVerbHandler

在发送Request的Message时，使用Message id注册了IAsyncCallback或IAsyncResult，而response的message和request message使用同一id。因此，在得到response message后，可根据id取出对应的callback

String messageId = message.getMessageId(); IAsyncCallback cb = MessagingService.getRegisteredCallback(messageId);if (cb != null) { cb.response(message);}else{ IAsyncResult ar = MessagingService.getAsyncResult(messageId); if (ar != null){ar.result(message);}}

5. message write 线程

MessagingService.sendOneWay 中得到对应的OutboundTcpConnection, 将消息放到OutboundTcpConnection的队列中,由OutboundTcpConnection向target address的7000端口发送.

• 每个OutboundTcpConnection 对应一个线程，循环消费队列的消息
• 每个InetAddress to 对应两个OutboundTcpConnection(RESPONSE_STAGE, GOSSIP_STAGE类型的message使用一个通道，其他类型使用另外一个通道)

while (true) { ByteBuffer bb = take(); if (bb == CLOSE_SENTINEL) { disconnect(); continue; } if (socket != null || connect()) writeConnected(bb); }private boolean connect() { if (logger.isDebugEnabled()) logger.debug("attempting to connect to " + endpoint); long start = System.currentTimeMillis(); while (System.currentTimeMillis() < start + DatabaseDescriptor.getRpcTimeout()) { try { // zero means 'bind on any available port.' socket = new Socket(endpoint, DatabaseDescriptor.getStoragePort(), FBUtilities.getLocalAddress(), 0); socket.setTcpNoDelay(true); output = new DataOutputStream(socket.getOutputStream()); return true; } catch (IOException e) { socket = null; if (logger.isTraceEnabled()) logger.trace("unable to connect to " + endpoint, e); try { Thread.sleep(OPEN_RETRY_DELAY); } catch (InterruptedException e1) { throw new AssertionError(e1); } } } return false; }

 

由以上代码可以看出，对单个Message，如果在RpcTimeout(2s)内没有连接上对方，则被直接丢弃.在2s内，每隔100ms会retry一次

 

总结

综上，网络端通信有如下几类线程:

1. 监听线程 SocketThread(7000) "ACCEPT-" + localEp
2. 每个socket(remote address)对应一个IncomingTcpConnection线程，保持长连接
3. Message Deserialization 线程
4. Message handler 线程池，不同类型的handler具有不同的线程池大小，读请求是8，写请求是32，处理响应是2
5. 每个remote address 对应两个OutboundTcpConnection线程，保持长连接

设计特点

• socket读、写线程分离，使用不同的线程来分别监听，读，写
• 异步响应。所有sendRR(request/response)中的方法都是异步响应, 由回调IAsyncCallback或者IAsyncResult来得到异步响应.异步响应通过使用Message id 注册callback来实现
• 不同类型的Message在不同的线程池中处理(stage对应线程池类型, verb对应handler)
• 不同类型的Message有header来区分，而不是使用多个子类, 按层次对message进行包装

         ----------------

          payload: RowMutation(...)        

         ----------------

           payload: Message[header + payload]

         ----------------

           payload:Comm Protocol [magic number + comm head + size + payload]

         ----------------

           wire payload