《Windows网络与通信程序设计》读书笔记－－－－可伸缩服务器系统设计实例

一个基于IOCP服务器的设计――CIOCPServer类

 

CIOCPServer类的总体结构：

CIOCPServer类工作主要由一个监听线程_ListenThreadProc和一个或多个工作线程_WorkerThreadProc负责。

 

监听线程_ListenThreadProc的工作：

监听线程负责初始投递几个异步的AcceptEx I/O，创建指定数目的工作线程，另外还负责补充投递异步AcceptEx I/O和检测恶意连接、通知工作线程退出的工作。

注意：投递AcceptEx I/O的工作只由监听线程负责。

 

工作线程_WorkerThreadProc的工作：

工作线程负责调用GetQueuedCompletionStatus等待异步I/O的完成，并调用HandleIO函数根据相应的异步I/O类型作出相应的处理，并投递相应的Read I/O和Write I/O，通知监听线程补充相应的AcceptEx I/O。

 

几点需要关注的：

1、 关于检测恶意连接的功能：

如果为AcceptEx函数提供了接收缓冲区，AcceptEx投递的重叠操作直到接受到连接并且讲到数据之后才会返回。但是有的恶意客户仅不断地调用connect函数连接服务器，既不发送数据，也不关闭连接，就会造成AcceptEx投递的大量重叠操作不能返回，为了满足客户的需求，服务器不得不再投递更多的接受I/O，占用了大量的系统资源。

为了避免这个事件发生，服务器应该记录所有AcceptEx投递的未决I/O，在线程中定时遍历它们，对每个客户套接字(可能还没有客户连接)以SO_CONNCET_TIME为参数调用getsockopt函数，检查连接建立了多长时间（如果建立了连接），如果时间过长，就将连接关闭。

2、关于内存资源管理的：

为了避免频繁地申请、释放内存(造成很多的内存碎片？)，CIOCPServer类使用内存池管理缓冲区对象和客户上下文对象使用的内存。具体情况是，使用指针保存所有空闲的内存块，形成空闲列表。当申请内在时如果这个指针不为NULL，就从空闲列表中取一个来使用，如果取完了(空闲链表为空)，再真正申请内存。释放内存时，如果没有超过指定上限，就仅简单地将要释放的内存添加到空闲链表即可。

另外一点就是关于CIOCPBuffer中buf成员指向的地方，其实就是为一个CIOCPBuffer申请内存分配时，紧贴CIOCPBuffer后面多分配出来的那一块大小为BUFFER_SIZE的内存，这种分配方案很有特色，木有见过，详细见代码。

3、关于包重新排序的问题：

虽然使用I/O完成端口的操作总会按照它们被提交的顺序完成，但是线程调度问题可能会导致关联到完成端口的实际工作不按正常顺序完成。例如，有两个I/O工作线程，应该接收“字节块1，字节块2，字节块3”，但是你可能以错误顺序接收到了这3个字节块：“字节块2，字节块1，字节块3”。这也意味着通过在完成端口投递发送请求发送数据时，数据实际会以错误的方式发送。书上关于此问题的一个简单的解决方案是向提交的缓冲区对象中添加序列号，如果缓冲区序列号是连续，就处理缓冲区中的数据。这意味着，有着错误序号的缓冲区要被保存下来，以便今后使用。另外，大多数网络协议都是基于封包的，这里开始的X个字节描述协议头，协议头包含的信息说明了整个封包到底有多大。服务器可以读协议头，计算出还需要读多少数据，然后继续读后面的数据，直到得到完整的封包。当服务器一次仅做一个异步读调用时，这工作得很好。但是，如果想要发挥IOCP服务器的全部潜力，就应该有多个未决的异步读操作等待数据的到来。这意味着，多个异步读操作不按顺序完成，未决读I/O返回的字节流不能按顺序处理，接收到字节流可能组合成正确的封包，也可能组合成错误的封包。所以，必须要为提交的读I/O分配序列号。

 

总结：断断续续敲了差不多一天的时间，终于将这个简单的实例敲完了。边敲边结合书本的话和代码一起理解，感觉书上这个简单的示例还是不错的，内存池管理技术、分配I/O包读序号、检测恶意连接等还是很值得借鉴，最重要是对如何使用IOCP机制有了一定的感觉。另外很久没用VS2005了，今天用它来跟踪调试，感觉很强大~~~

书上源代码：

CIOCPServer.h文件

/****************************************************** * CIOCPServer.h文件 * 主要关于per-handle和per-I/O和CIOCPServer类的声明 * ****************************************************/#ifndefCIOCP_H_#defineCIOCP_H_#include <winsock2.h>#include <windows.h>#include <Mswsock.h>#include<cstdio>#pragma comment(lib,"ws2_32.lib") #define BUFFER_SIZE1024*4//I/O请求的缓冲区的大小#defineMAX_THREAD2//I/O服务线程的数量//缓冲区对象//这是per-I/O数据struct CIOCPBuffer{WSAOVERLAPPED ol ;SOCKET sClient ;//AcceptEx接收的客户方套接字char *buff ;//I/O操作使用的缓冲区int nLen ;//buff缓冲区（使用的）大小ULONG nSequenceNumber ;//此I/O的序列号int nOperation ;//操作类型#define OP_ACCEPT1#define OP_WRITE2#define OP_READ3CIOCPBuffer *pNext ;} ;//客户上下文对象//这是per-Handle数据。它包含了一个套接字的信息struct CIOCPContext{SOCKETs ;//套接字句柄SOCKADDR_INaddrLocal ;//连接的本地地址SOCKADDR_IN addrRemote ;//连接的远程地址BOOLbClosing ;//套接字是否关闭int nOutstandingRecv ;//此套接字上抛出的重叠操作的数量int nOutstandingSend ;ULONG nReadSequence ;//安排给接收的下一个序列号ULONG nCurrentReadSequence ;//当前要读的序列号CIOCPBuffer *pOutOfOrderReads ;//记录没有按顺序完成的读I/OCRITICAL_SECTION Lock ;//保护这个结构CIOCPContext *pNext ;} ;class CIOCPServer//处理线程{public :CIOCPServer() ;~CIOCPServer() ;//开始服务BOOL Start(int nPort = 4567,int nMaxConnections = 2000,int nMaxFreeBuffers = 200,int nMaxFreeContexts = 100,int nInitialReads = 4)  ; //停止服务void Shutdown() ;//关闭一个连接和关闭所有连接void CloseAConnection(CIOCPContext *pContext) ;void CloseAllConnections() ;//取得当前的连接数量ULONG GetCurrentConnection(){return m_nCurrentConnection ;} //向指定客户发送文本BOOL SendText(CIOCPContext *pContext,char *pszText,int nLen) ;protected://申请和释放缓冲区对象CIOCPBuffer *AllocateBuffer(int nLen) ;void ReleaseBuffer(CIOCPBuffer *pBuffer) ;//申请和释放套接字上下文CIOCPContext *AllocateContext(SOCKET s) ;void ReleaseContext(CIOCPContext *pContext) ;//释放空闲缓冲区对象列表和空闲上下文对象列表void FreeBuffers() ;void FreeContexts() ;//向连接列表中添加一个连接BOOL AddAConnection(CIOCPContext *pContext) ;//插入和移除未决的接受请求BOOL InsertPendingAccept(CIOCPBuffer *pBuffer) ;BOOL RemovePendingAccept(CIOCPBuffer *pBuffer) ;//取得下一个要读取的CIOCPBuffer *GetNextReadBuffer(CIOCPContext *pContext,CIOCPBuffer *pBuffer) ;//投递接受I/O、发送I/O、接收I/OBOOL PostAccept(CIOCPBuffer *pBuffer) ;BOOL PostSend(CIOCPContext *pContext,CIOCPBuffer *pBuffer) ;BOOL PostRecv(CIOCPContext *pContext,CIOCPBuffer *pBuffer) ;//I/O处理函数void HandleIO(DWORD dwKey,CIOCPBuffer *pBuffer ,DWORD dwTrans,int nError) ;//事件通知函数,虚函数//建立了一个新的连接virtual void OnConnectionEstablished(CIOCPContext *pContext,CIOCPBuffer *pBuffer) ;//一个连接关闭virtual void OnConnectionClosing(CIOCPContext *pContext,CIOCPBuffer *pBuffer) ;//在一个连接上发生了错误virtual void OnConnectionError(CIOCPContext *pContext,CIOCPBuffer *pBuffer,int Error) ;//在一个连接上的读操作完成virtual void OnReadCompleted(CIOCPContext *pContext,CIOCPBuffer *pBuffer) ;//在一个连接上的写操作完成virtual void OnWriteCompleted(CIOCPContext *pContext,CIOCPBuffer *pBuffer) ;protected ://记录空闲结构信息CIOCPBuffer *m_pFreeBufferList ;//空闲的缓冲区链表CIOCPContext *m_pFreeContextList ;//空闲的套接字上下文链表int m_nFreeBufferCount ;int m_nFreeContextCount ;CRITICAL_SECTION m_FreeBufferListLock ;CRITICAL_SECTION m_FreeContextListLock ;//记录抛出的Accept请求CIOCPBuffer *m_pPendingAccepts ;//抛出的请求列表,未决链表long m_nPendingAcceptCount ;CRITICAL_SECTION m_PendingAcceptsLock ;//记录连接列表CIOCPContext *m_pConnectionList ;//已经连接的链表int m_nCurrentConnection ;CRITICAL_SECTION m_ConnectionListLock ;//用于投递Accept请求HANDLE m_hAcceptEvent ;HANDLE m_hRepostEvent ;//有反应LONG m_nRepostCount ;int m_nPort ;//服务器监听的端口int m_nInitialAccepts ;//初始投递的异步接受I/Oint m_nInitialReads ;//初始投递的异步Read I/Oint m_nMaxAccepts ;int m_nMaxSends ;int m_nMaxFreeBuffers ;int m_nMaxFreeContexts ;int m_nMaxConnections ;HANDLE m_hListenThread ;//监听线程HANDLE m_hCompletion ;//完成端口句柄SOCKETm_sListen ;//监听套接字句柄LPFN_ACCEPTEX m_lpfnAcceptEx ;//AcceptEx函数的地址LPFN_GETACCEPTEXSOCKADDRS m_lpfnGetAcceptExSockaddrs ; //GetAcceptExSockaddrs函数地址BOOL m_bShutDown ;BOOL m_bServerStarted ;private ://线程函数static DWORD WINAPI _ListenThreadProc(LPVOID lpParam) ;static DWORD WINAPI _WorkerThreadProc(LPVOID lpParam) ;} ;#endif

CIOCPServer.cpp文件

/**************************************************************** * CIOCPServer.cpp * CIOCPServer类接口的实现 * **************************************************************/#include"CIOCPServer.h"//初始化CIOCPServer::CIOCPServer(){//列表m_pFreeBufferList = NULL ;m_pFreeContextList = NULL ;m_pPendingAccepts = NULL ;m_pConnectionList = NULL ;m_nFreeBufferCount = 0 ;m_nFreeContextCount = 0 ;m_nPendingAcceptCount = 0 ;m_nCurrentConnection = 0 ;InitializeCriticalSection(&m_FreeBufferListLock) ;InitializeCriticalSection(&m_FreeContextListLock) ;InitializeCriticalSection(&m_PendingAcceptsLock) ;InitializeCriticalSection(&m_ConnectionListLock) ;//Accpet请求m_hAcceptEvent = CreateEvent(NULL,FALSE,FALSE,NULL) ;m_hRepostEvent = CreateEvent(NULL,FALSE,FALSE,NULL) ;m_nRepostCount = 0 ;m_nPort = 4567 ;m_nInitialAccepts = 10 ;m_nInitialReads = 4 ;m_nMaxAccepts = 100 ;m_nMaxSends = 20 ;m_nMaxFreeBuffers = 200 ;m_nMaxFreeContexts = 100 ;m_nMaxConnections = 2000 ;m_hListenThread = NULL ;m_hCompletion = NULL ;m_sListen = INVALID_SOCKET ;m_lpfnAcceptEx = NULL ;m_lpfnGetAcceptExSockaddrs = NULL ;m_bShutDown = FALSE ;m_bServerStarted = FALSE ;//初始化WS2_32.DLLWSADATA wsaData ;WORD sockVersion = MAKEWORD(2,2) ;WSAStartup(sockVersion,&wsaData) ;}//析构函数CIOCPServer::~CIOCPServer(){Shutdown() ;if(m_sListen != INVALID_SOCKET){closesocket(m_sListen);}if(m_hListenThread != NULL){CloseHandle(m_hListenThread);}CloseHandle(m_hRepostEvent);CloseHandle(m_hAcceptEvent);DeleteCriticalSection(&m_FreeBufferListLock) ;DeleteCriticalSection(&m_FreeContextListLock) ;DeleteCriticalSection(&m_PendingAcceptsLock) ;DeleteCriticalSection(&m_ConnectionListLock) ;WSACleanup() ;}//申请缓冲区对象的函数CIOCPBuffer* CIOCPServer::AllocateBuffer(int nLen){CIOCPBuffer *pBuffer = NULL ;if(nLen > BUFFER_SIZE)//超过最大缓冲区的长度{return NULL ;}//为缓冲区对象申请内存EnterCriticalSection(&m_FreeBufferListLock) ;if(NULL == m_pFreeBufferList)//(缓冲区)内存池为空，申请新内存{//进程的默认堆,那个BUFFER_SIZE的空间就是下面pBuffer->buff指向的pBuffer = (CIOCPBuffer *)HeapAlloc(GetProcessHeap(),HEAP_ZERO_MEMORY,sizeof(CIOCPBuffer)+BUFFER_SIZE) ; }else//从内存池中取一块来使用{pBuffer = m_pFreeBufferList ;m_pFreeBufferList = m_pFreeBufferList->pNext ;pBuffer->pNext = NULL ;m_nFreeBufferCount-- ;}LeaveCriticalSection(&m_FreeBufferListLock) ;//初始化新的缓冲区对象if(pBuffer != NULL){//我们让缓冲区对象中的buff成员指向的内存直接位于CIOCPBuffer对象之后，这样便于管理(书上原话)pBuffer->buff = (char*)(pBuffer+1) ;  pBuffer->nLen = nLen ;}return pBuffer ;}//释放缓冲区对象void CIOCPServer::ReleaseBuffer(CIOCPBuffer *pBuffer){EnterCriticalSection(&m_FreeBufferListLock) ;if(m_nFreeBufferCount <= m_nMaxFreeBuffers)//将要释放的内存添加到空闲到列表中{memset(pBuffer,0,sizeof(CIOCPBuffer)+BUFFER_SIZE) ;pBuffer->pNext = m_pFreeBufferList ;m_pFreeBufferList = pBuffer ;m_nFreeBufferCount++ ;}else//已经达到最大值，真正地释放内存{HeapFree(GetProcessHeap(),0,pBuffer) ;}LeaveCriticalSection(&m_FreeBufferListLock) ;}//申请套接字上下文CIOCPContext* CIOCPServer::AllocateContext(SOCKET s){CIOCPContext *pContext = NULL ;//为套接字上下文申请内存EnterCriticalSection(&m_FreeContextListLock) ;if(NULL == m_pFreeContextList)//套接字上下文池为空，申请新内存{pContext = (CIOCPContext *)HeapAlloc(GetProcessHeap(),HEAP_ZERO_MEMORY,sizeof(CIOCPContext)) ;InitializeCriticalSection(&pContext->Lock) ;//其中的临界区需要初始化}else//从套接字上下文池中取一块来使用{pContext = m_pFreeContextList ;m_pFreeContextList = m_pFreeContextList->pNext ;pContext->pNext = NULL ;m_nFreeContextCount-- ;}LeaveCriticalSection(&m_FreeContextListLock) ;//初始化新的套接字上下文对象if(pContext != NULL){pContext->s = s ;//待定先初始化套接字}returnpContext ;}//释放套接字上下文void CIOCPServer::ReleaseContext(CIOCPContext *pContext){if(pContext->s != INVALID_SOCKET){closesocket(pContext->s) ;}//首先释放（如果有的话）此套接字上的没有按顺序完成的读I/O的缓冲区CIOCPBuffer *pNext ;while(pContext->pOutOfOrderReads != NULL){pNext = pContext->pOutOfOrderReads->pNext ;ReleaseBuffer(pContext->pOutOfOrderReads) ;pContext->pOutOfOrderReads = pNext ;}EnterCriticalSection(&m_FreeContextListLock) ;if(m_nFreeContextCount <= m_nMaxFreeContexts)//添加到空闲到列表{CRITICAL_SECTION cstmp = pContext->Lock ;//先将关键代码段变量保存到一个临时变量中,为了再分配时使用 memset(pContext,0,sizeof(CIOCPContext)) ;//将要释放的上下文对象初始化为0//再放回关键代码段变量，将要释放的上下文对象添加到空闲列表的表头pContext->Lock = cstmp ;pContext->pNext = m_pFreeContextList ;m_pFreeContextList = pContext ;m_nFreeContextCount++ ;//更新计数}else{DeleteCriticalSection(&pContext->Lock) ;HeapFree(GetProcessHeap(),0,pContext) ;}LeaveCriticalSection(&m_FreeContextListLock) ;}//释放空闲缓冲区对象列表void CIOCPServer::FreeBuffers(){EnterCriticalSection(&m_FreeBufferListLock) ;CIOCPBuffer *pBuffer = m_pFreeBufferList  ;CIOCPBuffer *pNextBuffer = NULL ;while(pBuffer != NULL){pNextBuffer = pBuffer->pNext ;HeapFree(GetProcessHeap(),0,pBuffer) ;pBuffer = pNextBuffer ;}m_nFreeBufferCount = 0 ;m_pFreeBufferList = NULL ;LeaveCriticalSection(&m_FreeBufferListLock) ;}//释放空闲上下文对象列表void CIOCPServer::FreeContexts(){EnterCriticalSection(&m_FreeContextListLock) ;CIOCPContext *pContext = m_pFreeContextList ;CIOCPContext *pNextContext = NULL ;while(pContext != NULL){pNextContext = pContext->pNext ;HeapFree(GetProcessHeap(),0,pContext) ;pContext = pNextContext ;}m_nFreeContextCount = 0 ;m_pFreeContextList = NULL ;LeaveCriticalSection(&m_FreeContextListLock) ;}//向连接列表中添加一个连接BOOL CIOCPServer::AddAConnection(CIOCPContext *pContext) {//向客户连接列表添加一个CIOCPContext对象EnterCriticalSection(&m_ConnectionListLock) ;if(m_nCurrentConnection <= m_nMaxConnections){//添加到表头pContext->pNext = m_pConnectionList ;m_pConnectionList = pContext ;//更新计数m_nCurrentConnection++ ;LeaveCriticalSection(&m_ConnectionListLock) ;return TRUE ;}LeaveCriticalSection(&m_ConnectionListLock) ;return FALSE ;}//关闭一个连接,需要添加将该套接字添加到空闲套接字上下文处void CIOCPServer::CloseAConnection(CIOCPContext *pContext) {//关闭一个连接EnterCriticalSection(&m_ConnectionListLock) ;CIOCPContext *pTest = m_pConnectionList ;if(pTest == pContext)//被删除的是头结点{m_pConnectionList = pContext->pNext ;m_nCurrentConnection-- ;}else{//前一个结点while(pTest != NULL && pTest->pNext != pContext){pTest = pTest->pNext ;}if(pTest != NULL){pTest->pNext = pContext->pNext ;m_nCurrentConnection-- ;}}LeaveCriticalSection(&m_ConnectionListLock) ;//然后关闭客户套接字EnterCriticalSection(&pContext->Lock) ;if(pContext->s != INVALID_SOCKET){closesocket(pContext->s) ;pContext->s = INVALID_SOCKET ;}pContext->bClosing = TRUE ;LeaveCriticalSection(&pContext->Lock) ;}//关闭所有连接void CIOCPServer::CloseAllConnections() {//遍历整个连接列表，关闭所有的客户套接字EnterCriticalSection(&m_ConnectionListLock) ;CIOCPContext *pContext = m_pConnectionList ;while(pContext != NULL){EnterCriticalSection(&pContext->Lock) ;if(pContext->s != INVALID_SOCKET){closesocket(pContext->s) ;pContext->s = INVALID_SOCKET ;}pContext->bClosing = TRUE ;LeaveCriticalSection(&pContext->Lock) ;pContext = pContext->pNext ;}m_pConnectionList = NULL ;m_nCurrentConnection = 0 ;LeaveCriticalSection(&m_ConnectionListLock) ;}//插入的接受请求BOOL CIOCPServer::InsertPendingAccept(CIOCPBuffer *pBuffer) {//将一个I/O缓冲区对象插入到m_pPendingAccepts表中EnterCriticalSection(&m_PendingAcceptsLock) ;if(NULL == m_pPendingAccepts){m_pPendingAccepts = pBuffer ;}else{pBuffer->pNext = m_pPendingAccepts ;m_pPendingAccepts = pBuffer ;}m_nPendingAcceptCount++ ;LeaveCriticalSection(&m_PendingAcceptsLock) ;return TRUE ;}//移除未决的请求BOOL CIOCPServer::RemovePendingAccept(CIOCPBuffer *pBuffer) {BOOL bResult = FALSE ;//移除一个未决的请求EnterCriticalSection(&m_PendingAcceptsLock) ;CIOCPBuffer *pTest = m_pPendingAccepts ;if(pTest == pBuffer){m_pPendingAccepts = pBuffer->pNext ;}else//不是表头{while(pTest != NULL && pTest->pNext != pBuffer){pTest = pTest->pNext ;}if(pTest != NULL){pTest->pNext = pBuffer->pNext ;bResult = TRUE;}}//更新计数if(bResult){m_nPendingAcceptCount-- ;}LeaveCriticalSection(&m_PendingAcceptsLock) ;return bResult ;}//返回服务器下一个需要读的对象CIOCPBuffer *CIOCPServer::GetNextReadBuffer(CIOCPContext *pContext,CIOCPBuffer *pBuffer){if(pBuffer != NULL){//如果与要读的下一个序列号相等，则读这块缓冲区if(pBuffer->nSequenceNumber == pContext->nCurrentReadSequence){return pBuffer ;}//如果不相等，则说明没有按顺序接收数据，将这块缓冲区保存到连接的pOutOfOrderReads列表中//列表中的缓冲区是按照其序列号从小到大的顺序排列的pBuffer->pNext = NULL ;CIOCPBuffer *ptr = pContext->pOutOfOrderReads ;CIOCPBuffer *pPre = NULL ;while(ptr != NULL){if(pBuffer->nSequenceNumber < ptr->nSequenceNumber){break ;}pPre = ptr ;ptr = ptr->pNext ;}if(NULL == pPre) //插入到表头{pBuffer->pNext = pContext->pOutOfOrderReads ;pContext->pOutOfOrderReads = pBuffer ;}else//应该插入到表的中间{pBuffer->pNext = pPre->pNext ;pPre->pNext = pBuffer->pNext ;}}//检查表头元素的序列号，如果与要读的序号一致，就将它从表中移除，返回给用户CIOCPBuffer *ptr = pContext->pOutOfOrderReads ;if(ptr != NULL && (ptr->nSequenceNumber == pContext->nCurrentReadSequence)){pContext->pOutOfOrderReads = ptr->pNext ;return ptr ;}return NULL ;}//在监听套接字上投递Accept请求,这是一个投递监听语法，所以传入的参数是CIOCPBuffer结构，无需套接字 BOOL CIOCPServer::PostAccept(CIOCPBuffer *pBuffer) {//设置I/O类型pBuffer->nOperation = OP_ACCEPT ;//投递此重叠I/ODWORD dwBytes ;pBuffer->sClient = WSASocket(AF_INET,SOCK_STREAM,0,NULL,0,WSA_FLAG_OVERLAPPED) ;BOOL b = m_lpfnAcceptEx(m_sListen,//所有投递监听请求的共享这个监听套接字pBuffer->sClient,pBuffer->buff,pBuffer->nLen-((sizeof(sockaddr_in)+16)*2),sizeof(sockaddr_in)+16,sizeof(sockaddr_in)+16,&dwBytes,&pBuffer->ol) ;if(!b && WSAGetLastError() != WSA_IO_PENDING){return FALSE ;}return TRUE ;}//投递发送I/O异步请求,这里需要用到一个CIOCPContext，用于指示在哪一个套接字上面投递发送I/OBOOL CIOCPServer::PostSend(CIOCPContext *pContext,CIOCPBuffer *pBuffer) {//跟踪投递的发送的数量，防止用户仅发送数据而不接收，导致服务器抛出大量发送操作if(pContext->nOutstandingSend > m_nMaxSends){return FALSE ;}//设置I/O类型，增加套接字上的重叠I/O计数pBuffer->nOperation = OP_WRITE ;//投递此重叠I/ODWORD dwBytes ;DWORD dwFlags = 0 ;WSABUF buf ;buf.buf = pBuffer->buff ;buf.len = pBuffer->nLen ;if(WSASend(pContext->s,&buf,1,&dwBytes,dwFlags,&pBuffer->ol,NULL) != NO_ERROR){if(WSAGetLastError() != WSA_IO_PENDING){return FALSE ;}}//增加套接字上的重叠I/O计数EnterCriticalSection(&pContext->Lock) ;pContext->nOutstandingSend++ ;LeaveCriticalSection(&pContext->Lock) ;return TRUE ;}//投递接收I/O异步请求，这里也需要用一个CIOCPContext结构BOOL CIOCPServer::PostRecv(CIOCPContext *pContext,CIOCPBuffer *pBuffer) {//设置I/O类型pBuffer->nOperation = OP_READ ; EnterCriticalSection(&pContext->Lock) ;pBuffer->nSequenceNumber = pContext->nReadSequence ;//设置序列号//投递此重叠I/ODWORD dwBytes ;DWORD dwFlags = 0 ;WSABUF buf ;buf.buf = pBuffer->buff ;buf.len = pBuffer->nLen ;if(WSARecv(pContext->s,&buf,1,&dwBytes,&dwFlags,&pBuffer->ol,NULL) != NO_ERROR){if(WSAGetLastError() != WSA_IO_PENDING){LeaveCriticalSection(&pContext->Lock) ;return FALSE ;}}//增加套接字上的重叠I/O计数和读序列号计数pContext->nOutstandingRecv++ ;pContext->nReadSequence++ ;LeaveCriticalSection(&pContext->Lock) ;return TRUE ;}//开始服务BOOL CIOCPServer::Start(int nPort,int nMaxConnections,int nMaxFreeBuffers,int nMaxFreeContexts,int nInitialReads){//检查服务是否已经启动if(m_bServerStarted){return FALSE ;}//保存用户参数m_nPort = nPort ;m_nMaxConnections = nMaxConnections ;m_nMaxFreeBuffers = nMaxFreeBuffers ;m_nMaxFreeContexts = nMaxFreeContexts ;m_nInitialReads = nInitialReads ;//初始化状态变量m_bShutDown = FALSE ;m_bServerStarted = TRUE ;//创建监听套接字，绑定到本地端口，进入监听模式m_sListen = WSASocket(AF_INET,SOCK_STREAM,0,NULL,0,WSA_FLAG_OVERLAPPED) ;SOCKADDR_IN si ;si.sin_family = AF_INET ;si.sin_port = ntohs(m_nPort) ;si.sin_addr.s_addr = INADDR_ANY ;if(bind(m_sListen,(sockaddr*)&si,sizeof(si)) == SOCKET_ERROR){m_bServerStarted = FALSE ;return FALSE ;}listen(m_sListen,200) ;//创建完成端口对象m_hCompletion = CreateIoCompletionPort(INVALID_HANDLE_VALUE,0,0,0) ;//加载扩展函数AcceptExGUID GuidAcceptEx = WSAID_ACCEPTEX ;DWORD dwBytes ;WSAIoctl(m_sListen,SIO_GET_EXTENSION_FUNCTION_POINTER,&GuidAcceptEx,sizeof(GuidAcceptEx),&m_lpfnAcceptEx,sizeof(m_lpfnAcceptEx),&dwBytes,NULL,NULL) ;//加载扩展函数GetAcceptExSockaddrsGUID GuidGetAcceptExSockaddrs = WSAID_GETACCEPTEXSOCKADDRS ;WSAIoctl(m_sListen,SIO_GET_EXTENSION_FUNCTION_POINTER,&GuidGetAcceptExSockaddrs,sizeof(GuidGetAcceptExSockaddrs),&m_lpfnGetAcceptExSockaddrs,sizeof(m_lpfnGetAcceptExSockaddrs),&dwBytes,NULL,NULL);//将监听套接字关联到完成端口，注意，这里为它传递的CompletionKey为0CreateIoCompletionPort((HANDLE)m_sListen,m_hCompletion,(DWORD)0,0) ;//注册FD_ACCEPT事件//如果投递的AcceptEx I/O不够，线程会接收到FD_ACCEPT网络事件，说明应该投递更多的AcceptEx I/OWSAEventSelect(m_sListen,m_hAcceptEvent,FD_ACCEPT) ;//创建监听线程m_hListenThread = CreateThread(NULL,0,_ListenThreadProc,this,0,NULL) ;return TRUE ;}//监听者线程DWORD WINAPI CIOCPServer::_ListenThreadProc(LPVOID lpParam){CIOCPServer *pThis = (CIOCPServer *)lpParam ;//先在监听套接字上投递几个Accept I/OCIOCPBuffer *pBuffer ;for(int i = 0 ; i < pThis->m_nInitialAccepts ; ++i){pBuffer = pThis->AllocateBuffer(BUFFER_SIZE);if(NULL == pBuffer){return -1 ;}pThis->InsertPendingAccept(pBuffer);pThis->PostAccept(pBuffer);}//构建事件对象数组，以便在上面调用WSAWaitForMultipleEvents函数,前两个为事件对象，后面为线程对象HANDLE hWaitEvents[2+MAX_THREAD] ;int nEventCount = 0 ;hWaitEvents[nEventCount++] = pThis->m_hAcceptEvent ;//AcceptEx I/O用完了就触发这个事件hWaitEvents[nEventCount++] = pThis->m_hRepostEvent ;//AcceptEx I/O完成通知，每完成一个就触发一次//创建指定数量的工作线程在完成端口上处理I/Ofor(i = 0 ; i < MAX_THREAD ; ++i){hWaitEvents[nEventCount++] = CreateThread(NULL,0,_WorkerThreadProc,pThis,0,NULL) ;}//下面进入无限循环，处理事件对象数组中的事件while(TRUE){int nIndex = WSAWaitForMultipleEvents(nEventCount,hWaitEvents,FALSE,60*1000,FALSE) ;//首先检查是否要停止服务if(pThis->m_bShutDown || nIndex == WSA_WAIT_FAILED){//关闭所有连接pThis->CloseAllConnections() ;Sleep(0) ; //给I/O工作线程一个执行的机会,放弃自己剩余的CPU时间，但仍然处于就绪状态//关闭监听套接字closesocket(pThis->m_sListen) ;pThis->m_sListen = INVALID_SOCKET ;Sleep(0) ; //给I/O工作线程一个执行的机会//通知所有I/O处理线程退出for(int i = 2; i < MAX_THREAD + 2 ; ++i){PostQueuedCompletionStatus(pThis->m_hCompletion,-1,0,NULL) ; //第三个参数句柄唯一数据}//等待I/O处理线程退出,倒数第二个参数为真，表示等待所有线程退出WaitForMultipleObjects(MAX_THREAD,&hWaitEvents[2],TRUE,5*1000) ;for(i = 2 ; i < MAX_THREAD + 2 ; ++i){CloseHandle(hWaitEvents[i]) ;}CloseHandle(pThis->m_hCompletion) ;pThis->FreeBuffers() ;pThis->FreeContexts() ;//ExitThread(0) ;return 0 ; //代替为了ExitThread}//定时检查所有未返回的AcceptEx I/O的连接建立了多长时间if(nIndex == WSA_WAIT_TIMEOUT){pBuffer = pThis->m_pPendingAccepts ;while(pBuffer != NULL){int nSeconds ;int nLen = sizeof(nSeconds) ;//取得连接建立的时间getsockopt(pBuffer->sClient,SOL_SOCKET,SO_CONNECT_TIME,(char*)&nSeconds,&nLen) ;//如果超过2分钟客户还不发送初始数据，就让这个客户go awayif(nSeconds != -1 && nSeconds > 2 * 60){closesocket(pBuffer->sClient) ;pBuffer->sClient = INVALID_SOCKET ;}pBuffer = pBuffer->pNext ;}}else{nIndex = nIndex - WAIT_OBJECT_0 ;WSANETWORKEVENTS ne ;int nLimit = 0 ;if(0 == nIndex)//m_hAcceptEvent事件对象触发，说明投递的Accpet请求不够，需要增加{WSAEnumNetworkEvents(pThis->m_sListen,hWaitEvents[nIndex],&ne) ;if(ne.lNetworkEvents & FD_ACCEPT){nLimit = 50 ; //增加的个数，这里设为50个}}else if(1 == nIndex) //m_hRepostEvent事件对象触发，说明处理I/O的线程接受到新的客户{nLimit = InterlockedExchange(&pThis->m_nRepostCount,0) ;}else if(nIndex > 1) //I/O服务线程退出，说明有错误发生, 关闭服务器{pThis->m_bShutDown = TRUE ;continue ;}//投递nLimit个AccpetEx I/O请求,用了多少个就补充多少个int i = 0 ;while(i++ < nLimit && pThis->m_nPendingAcceptCount < pThis->m_nMaxAccepts){pBuffer = pThis->AllocateBuffer(BUFFER_SIZE) ;pThis->InsertPendingAccept(pBuffer) ;pThis->PostAccept(pBuffer) ;}}}return 0 ;}//停止服务void CIOCPServer::Shutdown() {if(!m_bServerStarted){return ;}//通知监听线程，马上停止服务m_bShutDown = TRUE ;SetEvent(m_hAcceptEvent) ;//等待监听线程退出WaitForSingleObject(m_hListenThread,INFINITE) ;CloseHandle(m_hListenThread) ;m_hListenThread = NULL ;m_bServerStarted = FALSE ;}//工作线程DWORD CIOCPServer::_WorkerThreadProc(LPVOID lpParam){#ifdef_DEBUGOutputDebugString("WorkerThread 启动...\n") ;#endifCIOCPServer *pThis = (CIOCPServer *)lpParam ;CIOCPBuffer  *pBuffer ;DWORD dwKey ;DWORD dwTrans ;LPOVERLAPPED lpol ;while(TRUE){//在关联到此完成端口的所有套接字上等待I/O完成BOOL bOK = GetQueuedCompletionStatus(pThis->m_hCompletion,&dwTrans,(LPDWORD)&dwKey,(LPOVERLAPPED*)&lpol,WSA_INFINITE) ;if(-1 == dwTrans)//传送的字节为负，传送完毕{#ifdef_DEBUGOutputDebugString("WorkerThread 退出\n") ;#endif//ExitThread(0) ;return 0 ; //代替ExitThread}//根据结构体的成员推算出结构体本身的指针pBuffer = CONTAINING_RECORD(lpol,CIOCPBuffer,ol) ;  //可以用强制类型转换做到，不过结构中的ol必须位于第一个成员int nError = NO_ERROR ;if(!bOK)//在此套接字上有错误发生,这一段没看明白作用{SOCKET s ;if(pBuffer->nOperation == OP_ACCEPT){s = pThis->m_sListen ;}else{if(0 == dwKey)//退出键{break ;}s = ((CIOCPContext *)dwKey)->s ;//有两种情况，有一种是0，另外一种就是句柄唯一数据}DWORD dwFlags = 0 ;if(!WSAGetOverlappedResult(s,&pBuffer->ol,&dwTrans,FALSE,&dwFlags)){nError = WSAGetLastError() ;}}pThis->HandleIO(dwKey,pBuffer,dwTrans,nError) ;}#ifdef _DEBUGOutputDebugString("Worker Thread 退出\n") ;#endifreturn 0 ;}//I/O处理函数void CIOCPServer::HandleIO(DWORD dwKey,CIOCPBuffer *pBuffer ,DWORD dwTrans,int nError) {CIOCPContext *pContext = (CIOCPContext *)dwKey ;#ifdef _DEBUGOutputDebugString("HandleIO..\n") ;#endif//1、首先减少套接上的未决I/O计数if(pContext != NULL){EnterCriticalSection(&pContext->Lock) ;if(pBuffer->nOperation == OP_READ){pContext->nOutstandingRecv-- ;}else if(pBuffer->nOperation == OP_WRITE){pContext->nOutstandingSend-- ;}LeaveCriticalSection(&pContext->Lock) ;//2、检查套接字是否已经被我们关闭if(pContext->bClosing){#ifdef _DEBUGOutputDebugString("检查到套接字已经被我们关闭\n") ;#endifif(pContext->nOutstandingRecv == 0 && pContext->nOutstandingSend == 0){ReleaseContext(pContext) ;}//释放已经关闭套接字的未决I/OReleaseBuffer(pBuffer) ;return  ;}}else//dwKey为0，证明是m_sListen套接字的per-handle数据，证明一个AcceptEx I/O被处理{RemovePendingAccept(pBuffer) ;//没看懂}//3检查套接字上发生的错误，如果有的话,通知用户，然后关闭套接字if(nError != NO_ERROR){if(pBuffer->nOperation != OP_ACCEPT){OnConnectionError(pContext,pBuffer,nError) ;CloseAConnection(pContext) ;if(pContext->nOutstandingRecv == 0 && pContext->nOutstandingSend == 0){ReleaseContext(pContext) ;}#ifdef_DEBUGOutputDebugString(" 检查到客户套接字上发生错误\n") ;#endif}else//在监听套接字上发生错误，也就是监听套接字处理的客户出错了{//客户端出错，释放I/O缓冲区if(pBuffer->sClient != INVALID_SOCKET){closesocket(pBuffer->sClient) ;pBuffer->sClient = INVALID_SOCKET ;}#ifdef _DEBUGOutputDebugString(" 检查到监听套接字上发生错误\n") ;#endif}ReleaseBuffer(pBuffer) ;return ;}//开始处理if(pBuffer->nOperation == OP_ACCEPT){if(0 == dwTrans){#ifdefDEBUG OutDebugString(" 监听套接字上客户端关闭\n") ;#endif if(pBuffer->sClient != INVALID_SOCKET){closesocket(pBuffer->sClient) ;pBuffer->sClient = INVALID_SOCKET ;}}else{//为新接受的连接申请客户上下文对象CIOCPContext *pClient = AllocateContext(pBuffer->sClient) ;if(pClient != NULL){if(AddAConnection(pClient)){//取得客户地址int nLocalLen,nRemoteLen ;LPSOCKADDR pLocalAddr,pRemoteAddr ;m_lpfnGetAcceptExSockaddrs(pBuffer->buff,pBuffer->nLen - ((sizeof(sockaddr_in)+16)*2),sizeof(sockaddr_in)+16,sizeof(sockaddr_in)+16,(SOCKADDR**)&pLocalAddr,&nLocalLen,(SOCKADDR **)&pRemoteAddr,&nRemoteLen) ;memcpy(&pClient->addrLocal,pLocalAddr,nLocalLen) ;memcpy(&pClient->addrRemote,pRemoteAddr,nRemoteLen) ;//关联新连接到完成端口对象CreateIoCompletionPort((HANDLE)pClient->s,m_hCompletion,(DWORD)pClient,0) ; //留意第三个参数，完成键//通知用户pBuffer->nLen = dwTrans ;OnConnectionEstablished(pClient,pBuffer) ;//向新连接投递几个Read请求，这些空间在套接字关闭或出错时释放for(int i = 0 ;i < 5 ; ++i){CIOCPBuffer *p = AllocateBuffer(BUFFER_SIZE) ;if(p != NULL){if(!PostRecv(pClient,p)){CloseAConnection(pClient) ;break ;}}}}else{//资源不足，关闭与客户的连接即可closesocket(pBuffer->sClient) ;pBuffer->sClient = INVALID_SOCKET ;}}//Accept请求完成，释放I/O缓冲区ReleaseBuffer(pBuffer) ;//通知监听线程继续再投递一个Accept请求InterlockedIncrement(&m_nRepostCount) ;SetEvent(m_hRepostEvent) ;}}else if(pBuffer->nOperation == OP_READ){if(0 == dwTrans)//对方关闭套接字{//先通知用户 pBuffer->nLen = 0 ;OnConnectionClosing(pContext,pBuffer) ;//再关闭连接CloseAConnection(pContext) ;//释放客户上下文和缓冲区对象if(pContext->nOutstandingRecv == 0 && pContext->nOutstandingSend == 0){ReleaseContext(pContext) ;}ReleaseBuffer(pBuffer) ;}else{pBuffer->nLen = dwTrans ;//按照I/O投递的顺序读取接收到的数据CIOCPBuffer *p = GetNextReadBuffer(pContext,pBuffer) ;while(p != NULL){OnReadCompleted(pContext,p) ;//通知用户//增加要读的序列号的值InterlockedIncrement((LONG*)&pContext->nCurrentReadSequence) ;//释放这个已完成的I/OReleaseBuffer(p) ;p = GetNextReadBuffer(pContext,NULL) ;}//继续投递一个新的接收请求pBuffer = AllocateBuffer(BUFFER_SIZE) ;if(NULL == pBuffer || !PostRecv(pContext,pBuffer)){CloseAConnection(pContext) ;}}}else if(pBuffer->nOperation == OP_WRITE){if(0 == dwTrans)//对方关闭套接字{//先通知用户pBuffer->nLen = 0 ;OnConnectionClosing(pContext,pBuffer) ;//再关闭连接CloseAConnection(pContext) ;//释放客户上下文和缓冲区对象if(pContext->nOutstandingRecv == 0 && pContext->nOutstandingSend == 0){ReleaseContext(pContext) ;}ReleaseBuffer(pBuffer) ;}else{//写操作完成，通知用户pBuffer->nLen = dwTrans ;OnWriteCompleted(pContext,pBuffer) ;//释放SendText函数申请的缓冲区ReleaseBuffer(pBuffer) ;}}}//发送数据BOOL CIOCPServer::SendText(CIOCPContext *pContext,char *pszText,int nLen){CIOCPBuffer *pBuffer = AllocateBuffer(nLen) ;if(pBuffer != NULL){memcpy(pBuffer->buff,pszText,nLen) ;return PostSend(pContext,pBuffer) ;}return FALSE ;}void CIOCPServer::OnConnectionEstablished(CIOCPContext *pContext, CIOCPBuffer *pBuffer){}void CIOCPServer::OnConnectionClosing(CIOCPContext *pContext, CIOCPBuffer *pBuffer){}void CIOCPServer::OnReadCompleted(CIOCPContext *pContext, CIOCPBuffer *pBuffer){}void CIOCPServer::OnWriteCompleted(CIOCPContext *pContext, CIOCPBuffer *pBuffer){}void CIOCPServer::OnConnectionError(CIOCPContext *pContext, CIOCPBuffer *pBuffer, int nError){}

main.cpp文件

////////////////////////////////////////////////// CIOCPServer类的测试程序#include "CIOCPServer.h"#include <stdio.h>#include <windows.h>class CMyServer : public CIOCPServer{public:void OnConnectionEstablished(CIOCPContext *pContext, CIOCPBuffer *pBuffer){printf(" 接收到一个新的连接（%d）： %s \n", GetCurrentConnection(), ::inet_ntoa(pContext->addrRemote.sin_addr));SendText(pContext, pBuffer->buff, pBuffer->nLen);}void OnConnectionClosing(CIOCPContext *pContext, CIOCPBuffer *pBuffer){printf(" 一个连接关闭！ \n" );}void OnConnectionError(CIOCPContext *pContext, CIOCPBuffer *pBuffer, int nError){printf(" 一个连接发生错误： %d \n ", nError);}void OnReadCompleted(CIOCPContext *pContext, CIOCPBuffer *pBuffer){SendText(pContext, pBuffer->buff, pBuffer->nLen);}void OnWriteCompleted(CIOCPContext *pContext, CIOCPBuffer *pBuffer){printf(" 数据发送成功！\n ");}};void main(){CMyServer *pServer = new CMyServer;// 开启服务if(pServer->Start()){printf(" 服务器开启成功... \n");}else{printf(" 服务器开启失败！\n");return;}// 创建事件对象，让ServerShutdown程序能够关闭自己HANDLE hEvent = ::CreateEvent(NULL, FALSE, FALSE, "ShutdownEvent");::WaitForSingleObject(hEvent, INFINITE);::CloseHandle(hEvent);// 关闭服务pServer->Shutdown();delete pServer;printf(" 服务器关闭 \n ");}