【Win32多线程】异步I/O技术(Overlapped I/O),避免使用多线程

讨论：1.激发的文件handles
      2.激发的event对象
      3.异步过程调用
      4.I/O completion ports(重要)适用于高负载服务器

----Win32文件操作函数
 
   win32中有上那个基本的函数用来执行I/O,他们是：
    CreateFile(),ReadFile(),WriteFile();
   没有哪个函数用来关闭文件，只要调用CloseHandle()即可。

CreateFile()可以用来打开各式各样的资源：包括文件，串口和并行口，Named pipes,Console.
它的原型如下：
HANDLE CreateFile(
  LPCTSTR lpFileName,                         // file name
  DWORD dwDesiredAccess,                      // access mode存取模式(读或写)
  DWORD dwShareMode,                          // share mode
  LPSECURITY_ATTRIBUTES lpSecurityAttributes, // SD
  DWORD dwCreationDisposition,                // how to create 如何产生
  DWORD dwFlagsAndAttributes,                 // file attributes 文件属性(使用overlapped I/O的关键)
  HANDLE hTemplateFile                        // handle to template file一个临时文件，将拥有全部的属性拷贝
);

C runtime library不能使用overlapped I/O.Overlapped I/O的基本型式是以ReadFile()和WriteFile()完成的。这两个函数的原型如下：
BOOL ReadFile(
  HANDLE hFile,                // handle to file欲读取的文件
  LPVOID lpBuffer,             // data buffer 接收数据的缓冲区
  DWORD nNumberOfBytesToRead,  // number of bytes to read 欲读取的字节数
  LPDWORD lpNumberOfBytesRead, // number of bytes read 实际读取的字节个数的地址
  LPOVERLAPPED lpOverlapped    // overlapped buffer指针，指向overlapped info
);

BOOL WriteFile(
  HANDLE hFile,                    // handle to file欲写的文件
  LPCVOID lpBuffer,                // data buffer存储数据的缓冲区
  DWORD nNumberOfBytesToWrite,     // number of bytes to write欲写入的字节个数
  LPDWORD lpNumberOfBytesWritten,  // number of bytes written实际写入的字节个数
  LPOVERLAPPED lpOverlapped        // overlapped buffer  指针，指向overlapped info
);

这两个函数很像C runtime 函数的fread 和fwrite()，差别在于最后一个参数lpOverlapped.如果第六个参数设为
FILE_FLAG_OVERLAPPED,则必须在lpOverlapped参数中提供一个指针，指向一个OVERLAPPED结构。

---OVERLAPPED结构
它执行两个重要的功能，第一，它像一把钥匙，用以识别一个目前正在进行的overlapped操作。第二，它在你和系统之间提供了一个共享区域，参数可以在该区域中双向传递。
typedef struct _OVERLAPPED {
    ULONG_PTR  Internal;
    ULONG_PTR  InternalHigh;
    DWORD  Offset;
    DWORD  OffsetHigh;
    HANDLE hEvent;
} OVERLAPPED;

由于OVERLAPPED结构的生命周期超越ReadFile()和WriteFile()函数。所以把这个结构放在局部变量不是安全的。应该放在heap上。

---被激发的File Handles

最简单的overlapped I/O类型，就是使用它自己文件handle作为同步机制。
首先，用FILE_FLAG_OVERLAPPED告诉Win32你不要使用默认的同步I/O。然后设立一个OVERLAPPED结构，其中内含“I/O请求”的所有必要的参数，并以此来识别这个“ I/O请求”，直到它完成为止。接下来，调用ReadFile()并以OVERLAPPED结构的地址作为最后一个参数。这时候，Win32会在后台处理你的请求。你的程序可以干其他的事情了。

   如果需要等待overlapped I/O的执行结果，作为WaitForMultipleObjects()的一部分，在handle数组中加上这个文件的handle.文件handle是一个核心对象。一旦操作完毕即被激发。当完成操作之后，

调用GetOverlappedResult()确定结果如何。
BOOL GetOverlappedResult(
  HANDLE hFile,                       // handle to file, pipe, or device
  LPOVERLAPPED lpOverlapped,          // overlapped structure 一个指针，指向OVERLAPPED结构
  LPDWORD lpNumberOfBytesTransferred, // bytes transferred表示真正被传送的字节
  BOOL bWait                          // wait option是否表示要等待操作完成
);

文件读取示例：

#define WIN32_LEAN_AND_MEAN#include <stdio.h>#include <stdlib.h>#include <windows.h>//// Constants//#define READ_SIZE       512//// Function prototypes//void CheckOsVersion();int main(){    BOOL rc;    HANDLE hFile;    DWORD numread;    OVERLAPPED overlap;    char buf[READ_SIZE];    char szPath[MAX_PATH];    CheckOsVersion();    GetWindowsDirectory(szPath, sizeof(szPath));    strcat(szPath, "\\WINHLP32.EXE");    // Open the file for overlapped reads    hFile = CreateFile( szPath,                    GENERIC_READ,                    FILE_SHARE_READ|FILE_SHARE_WRITE,                    NULL,                    OPEN_EXISTING,                    FILE_FLAG_OVERLAPPED,                    NULL                );    if (hFile == INVALID_HANDLE_VALUE)    {        printf("Could not open %s\n", szPath);        return -1;    }    // Initialize the OVERLAPPED structure    memset(&overlap, 0, sizeof(overlap));    overlap.Offset = 1500;    // Request the data    rc = ReadFile(                hFile,                buf,                READ_SIZE,                &numread,                &overlap            );    printf("Issued read request\n");    // Was the operation queued?    if (rc)    {        // The data was read successfully        printf("Request was returned immediately\n");    }    else    {        if (GetLastError() == ERROR_IO_PENDING)        {            // We could do something else for awhile here...            printf("Request queued, waiting...\n");            WaitForSingleObject(hFile, INFINITE);//这个调用其实是多余，因为GetOverlappedResult就可以等待                                                      overlapped操作的完成。            printf("Request completed.\n");            rc = GetOverlappedResult(                                    hFile,                                    &overlap,                                    &numread,                                    FALSE                                );            printf("Result was %d\n", rc);        }        else        {   // We should check for memory and quota   // errors here and retry. See the samples   // IoByEvnt and IoByAPC.            // Something went wrong            printf("Error reading file\n");        }    }    CloseHandle(hFile);    return EXIT_SUCCESS;}//// Make sure we are running under an operating// system that supports overlapped I/O to files.//void CheckOsVersion(){    OSVERSIONINFO   ver;    BOOL            bResult;    ver.dwOSVersionInfoSize = sizeof(OSVERSIONINFO);    bResult = GetVersionEx((LPOSVERSIONINFO) &ver);    if ( (!bResult) ||         (ver.dwPlatformId != VER_PLATFORM_WIN32_NT) )    {        fprintf(stderr, "IoByFile must be run under Windows NT.\n");  exit(EXIT_FAILURE);    }}

Q:Overlapped I/O总是异步的执行吗？
虽然你要求overlapped操作，但它并不一定就是overlapped。如果数据已经被放进cache中，或如果操作系统认为它可以很快速的取得那份数据，那么文件操作就会在ReadFile()返回之前完成。而ReadFile()将传回TRUE.这种情况下，文件handle处于激发状态，而对文件的操作可以视作就像overlapped一样。
  如果你要求一个文件操作为overlapped，而操作系统吧这个“操作请求”放到队列中等待执行，那么ReadFile()和WriteFile()都会传回FALSE表示失败。你必须调GetLastError()并确定它传回ERROR_IO_PENDING,那意味着“overlapped  I/O请求”被放进队列之中等待执行。GetLastError()也可能传回其他的值，例如ERROR_HANDLE_EOF，那么就代表一个错误。
OVERLAPPED结构体有能力处理64位的偏移值，因此面对巨大的文件也不怕。为了等待文件操作的完成，把文件的handle交给WaitForSingleObject().一旦overlapped操作完成，该文件的handle就会成为激发状态。

---被激发的Event对象
   用文件的handle作为激发机制，就没办法说出到底哪一个overlapped操作完成了。Win32有一个很好的解决办法。
OVERLAPPED结构中的最后一个栏位，是一个event handle.如果你使用文件handle作为激发对象，那么此栏设置为NULL.当使用event作为激发对象时，则赋值为event对象。系统核心会在overlapped操作

完成的时候，自动将此event对象给激发起来。由于每一个overlapped操作都有它自己独一无二的OVERLAPPED结构。所以每一个结构都有它独一无二的一个event对象，用来代表该操作。

Q:如何为overlapped I/O产生一个event对象？
你所使用的event对象必须是手动重置而非自动重置。如果使用自动重置，就有可能产生race condition,因为系统核心有可能在你有机会等待该event对象之前，先激发它。event对象的激发状态不能够保存的。于是这个event状态将遗失，而你的Wait...()函数永远不返回。但是一个手动重置的event，一旦激发，就一直处于激发状态。

 使用event对象搭配overlapped I/O，可以对同一个文件发出多个读取操作和多个写入操作，每一个操作有自己的event对象；然后在调用WaitForMultipleObjects()来等待其中之一完成。

使用Event和overlapped I/O实现文件读取示例:

/* Demonstrates how to use event objects instead of * file handles to signal multiple outstanding * overlapped operation on a file. */#define WIN32_LEAN_AND_MEAN#include <stdio.h>#include <stdlib.h>#include <windows.h>#include "MtVerify.h"//// Constants//#define MAX_REQUESTS    5#define READ_SIZE       512#define MAX_TRY_COUNT   5//// Function prototypes//int QueueRequest(int nIndex, DWORD dwLocation, DWORD dwAmount);void CheckOsVersion();//// Global variables//// Need to keep the events in their own array// so we can wait on them.HANDLE  ghEvents[MAX_REQUESTS];// Keep track of each individual I/O operationOVERLAPPED gOverlapped[MAX_REQUESTS];// Handle to the file of interest.HANDLE ghFile;// Need a place to put all this datachar gBuffers[MAX_REQUESTS][READ_SIZE];int main(){    int i;    BOOL rc;    char szPath[MAX_PATH];    CheckOsVersion();    GetWindowsDirectory(szPath, sizeof(szPath));    strcat(szPath, "\\WINHLP32.EXE");    // Open the file for overlapped reads    ghFile = CreateFile( szPath,                    GENERIC_READ,                    FILE_SHARE_READ|FILE_SHARE_WRITE,                    NULL,                    OPEN_EXISTING,                    FILE_FLAG_OVERLAPPED,                    NULL                );    if (ghFile == INVALID_HANDLE_VALUE)    {        printf("Could not open %s\n", szPath);        return -1;    }    for (i=0; i<MAX_REQUESTS; i++)    {        // Read some bytes every few K        QueueRequest(i, i*16384, READ_SIZE);    }    printf("QUEUED!!\n");    // Wait for all the operations to complete.    MTVERIFY( WaitForMultipleObjects(               MAX_REQUESTS, ghEvents, TRUE, INFINITE        ) != WAIT_FAILED );    // Describe what just happened.    for (i=0; i<MAX_REQUESTS; i++)    {        DWORD dwNumread;        rc = GetOverlappedResult(                                ghFile,                                &gOverlapped[i],                                &dwNumread,                                FALSE                            );        printf("Read #%d returned %d. %d bytes were read.\n",                    i, rc, dwNumread);        CloseHandle(gOverlapped[i].hEvent);    }    CloseHandle(ghFile);    return EXIT_SUCCESS;}/* * Call ReadFile to start an overlapped request. * Make sure we handle errors that are recoverable. * Properly set up the event object for this operation. */int QueueRequest(int nIndex, DWORD dwLocation, DWORD dwAmount){    int i;    BOOL rc;    DWORD dwNumread;    DWORD err;    MTVERIFY(        ghEvents[nIndex] = CreateEvent(                     NULL,    // No security                     TRUE,    // Manual reset - extremely important!                     FALSE,   // Initially set Event to non-signaled state                     NULL     // No name                    )    );    gOverlapped[nIndex].hEvent = ghEvents[nIndex];    gOverlapped[nIndex].Offset = dwLocation;    for (i=0; i<MAX_TRY_COUNT; i++)    {        rc = ReadFile(            ghFile,            gBuffers[nIndex],            dwAmount,            &dwNumread,            &gOverlapped[nIndex]        );        // Handle success        if (rc)        {            printf("Read #%d completed immediately.\n", nIndex);            return TRUE;        }        err = GetLastError();        // Handle the error that isn't an error. rc is zero here.        if (err == ERROR_IO_PENDING)        {            // asynchronous i/o is still in progress             printf("Read #%d queued for overlapped I/O.\n", nIndex);            return TRUE;        }        // Handle recoverable error        if ( err == ERROR_INVALID_USER_BUFFER ||             err == ERROR_NOT_ENOUGH_QUOTA ||             err == ERROR_NOT_ENOUGH_MEMORY )        {            Sleep(50);  // Wait around and try later            continue;        }        // Give up on fatal error.        break;    }    printf("ReadFile failed.\n");    return -1;}//// Make sure we are running under an operating// system that supports overlapped I/O to files.//void CheckOsVersion(){    OSVERSIONINFO   ver;    BOOL            bResult;    ver.dwOSVersionInfoSize = sizeof(OSVERSIONINFO);    bResult = GetVersionEx((LPOSVERSIONINFO) &ver);    if ( (!bResult) ||         (ver.dwPlatformId != VER_PLATFORM_WIN32_NT) )    {        fprintf(stderr, "IoByEvnt must be run under Windows NT.\n");  exit(EXIT_FAILURE);    }}

对于以Remote Access Service连接到一个服务器，现在可以记录这些操作过程，并做成overlapped I/O,然后在住消息循环中使用MsgWaitForMultipleObjects()以便在取得数据后有所反应。

-----------异步过程调用(Asynchronous Procedure Calls,APCs)
使用overlapped I/O配合event对象，会产生两个基础性问题。

第一，使用WaitForMultipleObjects(),你只能够等待最多MAXIMUM_WAIT_OBJECTS个对象。在Win32 SDK中，最大值为64。如果要等待64个以上的对象，就会出现问题。

第二，你必须不断根据“哪一个handle被激发”而计算如果反应。你必须分配表格和WaitForMultipleObjects()的handles数组结合起来。以上两个问题可以用异步过程调用来解决。
  只要使用“Ex”版本的ReadFile()和WriteFile()，就可以调用这个机制。这两个函数允许你指定一个额外的参数，是一个callback函数地址。当一个overlapped I/O完成时，系统应该调用该callback函数。这个callback函数称为I/O completion routine.

Q:一个I/O completion routine何时被调用？
  你的线程必须在所谓"alertable"状态下才行。如果线程不处于“alertable”状态，那么对I/O completion routine的调用会暂时被保存下来。因此，当一个线程终于进入“alertable”状态时，可能已经有一大堆储备的APCs等待被处理。

  如果线程因为下面五个函数处于等待状态，其“alertable”标记被设为TRUE,则该线程就是处于"alertable"状态：
  SleepEx(),WaitForSingleObjectEx();WaitForMultipleObjectsEx(),MsgWaitForMultipleObjectsEx();
SignalObjectAndWait();
只有在程序处于“alertable”状态下，APCs才会被调用。

I/O completion routine的形式：
VOID CALLBACK FileIOCompletionRoutine(
  DWORD dwErrorCode,                // completion code 0表示操作完成，ERROR_HANDLE_EOF表示操作已经到文件尾                                                  端。
  DWORD dwNumberOfBytesTransfered,  // number of bytes transferred真正被传输的字节数
  LPOVERLAPPED lpOverlapped         // I/O information buffer
);

 

Q：如何把一个用户自定义的数据传递给I/O completion routine？
使用APCs时，OVERLAPPED机构体重的hEvent位不需要用来放置一个event handle.hEvent栏位可以由程序员自由使用。最大的用途就是：首先配置一个结构，描述数据来自哪里，或是要对数据进行一些什么操作，然后将hEvent栏位设定为指向该结构。

overlapped I/O与APCs的应用示例：

/* Demonstrates how to use APC's (asynchronous * procedure calls) instead of signaled objects * to service multiple outstanding overlapped * operations on a file. */#define WIN32_LEAN_AND_MEAN#include <stdio.h>#include <stdlib.h>#include <windows.h>#include "MtVerify.h"//// Constants//#define MAX_REQUESTS    5#define READ_SIZE       512#define MAX_TRY_COUNT   5//// Function prototypes//void CheckOsVersion();int QueueRequest(int nIndex, DWORD dwLocation, DWORD dwAmount);//// Global variables//// Need a single event object so we know when all I/O is finishedHANDLE  ghEvent;// Keep track of each individual I/O operationOVERLAPPED gOverlapped[MAX_REQUESTS];// Handle to the file of interest.HANDLE ghFile;// Need a place to put all this datachar gBuffers[MAX_REQUESTS][READ_SIZE];int nCompletionCount;/* * I/O Completion routine gets called * when app is alertable (in WaitForSingleObjectEx) * and an overlapped I/O operation has completed. */VOID WINAPI FileIOCompletionRoutine(    DWORD dwErrorCode,  // completion code     DWORD dwNumberOfBytesTransfered,    // number of bytes transferred     LPOVERLAPPED lpOverlapped   // pointer to structure with I/O information     ){    // The event handle is really the user defined data    int nIndex = (int)(lpOverlapped->hEvent);    printf("Read #%d returned %d. %d bytes were read.\n",        nIndex,        dwErrorCode,        dwNumberOfBytesTransfered);    if (++nCompletionCount == MAX_REQUESTS)        SetEvent(ghEvent);  // Cause the wait to terminate}int main(){    int i;    char szPath[MAX_PATH];    CheckOsVersion();    // Need to know when to stop    MTVERIFY(        ghEvent = CreateEvent(                     NULL,    // No security                     TRUE,    // Manual reset - extremely important!                     FALSE,   // Initially set Event to non-signaled state                     NULL     // No name                    )    );    GetWindowsDirectory(szPath, sizeof(szPath));    strcat(szPath, "\\WINHLP32.EXE");    // Open the file for overlapped reads    ghFile = CreateFile( szPath,                    GENERIC_READ,                    FILE_SHARE_READ|FILE_SHARE_WRITE,                    NULL,                    OPEN_EXISTING,                    FILE_FLAG_OVERLAPPED,                    NULL                );    if (ghFile == INVALID_HANDLE_VALUE)    {        printf("Could not open %s\n", szPath);        return -1;    }    // Queue up a few requests    for (i=0; i<MAX_REQUESTS; i++)    {        // Read some bytes every few K        QueueRequest(i, i*16384, READ_SIZE);    }    printf("QUEUED!!\n");    // Wait for all the operations to complete.    for (;;)    {        DWORD rc;        rc = WaitForSingleObjectEx(ghEvent, INFINITE, TRUE );        if (rc == WAIT_OBJECT_0)            break;        MTVERIFY(rc == WAIT_IO_COMPLETION);    }    CloseHandle(ghFile);    return EXIT_SUCCESS;}/* * Call ReadFileEx to start an overlapped request. * Make sure we handle errors that are recoverable. */int QueueRequest(int nIndex, DWORD dwLocation, DWORD dwAmount){    int i;    BOOL rc;    DWORD err;    gOverlapped[nIndex].hEvent = (HANDLE)nIndex;    gOverlapped[nIndex].Offset = dwLocation;    for (i=0; i<MAX_TRY_COUNT; i++)    {        rc = ReadFileEx(            ghFile,            gBuffers[nIndex],            dwAmount,            &gOverlapped[nIndex],            FileIOCompletionRoutine        );        // Handle success        if (rc)        {            // asynchronous i/o is still in progress             printf("Read #%d queued for overlapped I/O.\n", nIndex);            return TRUE;        }        err = GetLastError();        // Handle recoverable error        if ( err == ERROR_INVALID_USER_BUFFER ||             err == ERROR_NOT_ENOUGH_QUOTA ||             err == ERROR_NOT_ENOUGH_MEMORY )        {            Sleep(50);  // Wait around and try later            continue;        }        // Give up on fatal error.        break;    }    printf("ReadFileEx failed.\n");    return -1;}//// Make sure we are running under an operating// system that supports overlapped I/O to files.//void CheckOsVersion(){    OSVERSIONINFO   ver;    BOOL            bResult;    ver.dwOSVersionInfoSize = sizeof(OSVERSIONINFO);    bResult = GetVersionEx((LPOSVERSIONINFO) &ver);    if ( (!bResult) ||         (ver.dwPlatformId != VER_PLATFORM_WIN32_NT) )    {        fprintf(stderr, "IoByAPC must be run under Windows NT.\n");  exit(EXIT_FAILURE);    }}

Q:如何把C++成员函数当做一个 I/O completion routine?
  一个C++成员函数不能拿来做一个I/O completion routine,除非它是static函数。如果它是static函数，它就没有this指针，也就不能直接调用那些需要this 指针的成员函数。
解决的方法是存储一个指针，指向用户自定义数据，然后经由此指针调用一个C++成员函数。由于static成员函数是类的一部分，还是可以调用private成员函数。与如何把C++成员函数当做一个线程函数类似。

另外一个I/O completion ports非常重要，对产生高效率的服务器很受欢迎，这里略，详细参考相关文献资料。