How to correctly read data when using epoll_wait

Question

I am trying to port to Linux an existing Windows C++ code that uses IOCP. Having decided to use epoll_wait to achieve high concurrency, I am already faced with a theoretical issue of when we try to process received data.

Imagine two threads calling epoll_wait, and two consequetives messages being received such that Linux unblocks the first thread and soon the second.

Example :

Thread 1 blocks on epoll_waitThread 2 blocks on epoll_waitClient sends a chunk of data 1Thread 1 deblocks from epoll_wait, performs recv and tries to process dataClient sends a chunk of data 2Thread 2 deblocks, performs recv and tries to process data.

Is this scenario conceivable ? I.e. can it occure ?

Is there a way to prevent it so to avoid implementing synchronization in the recv/processing code ?

score 4 · Accepted Answer · 2011-04-08 01:39:08Z

If you have multiple threads reading from the same set of epoll handles, I would recommend you put your epoll handles in one-shot level-triggered mode with EPOLLONESHOT. This will ensure that, after one thread observes the triggered handle, no other thread will observe it until you use epoll_ctl to re-arm the handle.

If you need to handle read and write paths independently, you may want to completely split up the read and write thread pools; have one epoll handle for read events, and one for write events, and assign threads to one or the other exclusively. Further, have a separate lock for read and for write paths. You must be careful about interactions between the read and write threads as far as modifying any per-socket state, of course.

If you do go with that split approach, you need to put some thought into how to handle socket closures. Most likely you will want an additional shared-data lock, and 'acknowledge closure' flags, set under the shared data lock, for both read and write paths. Read and write threads can then race to acknowledge, and the last one to acknowledge gets to clean up the shared data structures. That is, something like this:

void OnSocketClosed(shareddatastructure *pShared, int writer){  epoll_ctl(myepollhandle, EPOLL_CTL_DEL, pShared->fd, NULL);  LOCK(pShared->common_lock);  if (writer)    pShared->close_ack_w = true;  else    pShared->close_ack_r = true;  bool acked = pShared->close_ack_w && pShared->close_ack_r;  UNLOCK(pShared->common_lock);  if (acked)    free(pShared);}

So same as suggested by StitchedUp, this appears to be the best solution. However I ma sad that write events and read events can not be processed in // — Apr 7 '11 at 20:25
@charfeddine, you could have separate threads for write and read events, each using a different epoll handle with watches only on write or read, not both. This would allow you to parallelize write and read responses, although I tend to doubt it would make a big difference in performance. — Apr 7 '11 at 20:39
How is that please ? When I read the Linux Programmer manual, I only learn that the act of rearming the socket applies to the socket file descriptor as a whole. In my application this will make a major difference : server responses are not necessarily piloted by client requests : it is gonna be noticeable defect that server asynchronous broadcast information pause whe client sends something. — Apr 7 '11 at 22:40
Updated my answer with something perhaps a bit better attuned to what you need — Apr 8 '11 at 1:39
@charfeddine: you are not checking for errors. The second epoll_mod will (and does, I just tested) fail with EEXIST; you will not easily notice if you're attempting to send immediately before queueing up data for write, but you will never get a write-ready notification with this. — Apr 9 '11 at 3:00

StitchedUp 414 · Answer 2 · 2011-04-04 17:32:46Z

I'm assuming here that the situation you're trying to process is something like this:

You have multiple (maybe very many) sockets that you want to receive data from at once;

You want to start processing data from the first connection on Thread A when it is first received and then be sure that data from this connection is not processed on any other thread until you have finished with it in Thread A.

While you are doing that, if some data is now received on a different connection you want Thread B to pick that data and process it while still being sure that no one else can process this connection until Thread B is done with it etc.

Under these circumstances it turns out that using epoll_wait() with the same epoll fd in multiple threads is a reasonably efficient approach (I'm not claiming that it is necessarily the most efficient).

The trick here is to add the individual connections fds to the epoll fd with the EPOLLONESHOT flag. This ensures that once an fd has been returned from an epoll_wait() it is unmonitored until you specifically tell epoll to monitor it again. This ensures that the thread processing this connection suffers no interference as no other thread can be processing the same connection until this thread marks the connection to be monitored again.

You can set up the fd to monitor EPOLLIN or EPOLLOUT again using epoll_ctl() and EPOLL_CTL_MOD.

A significant benefit of using epoll like this in multiple threads is that when one thread is finished with a connection and adds it back to the epoll monitored set, any other threads still in epoll_wait() are immediately monitoring it even before the previous processing thread returns to epoll_wait(). Incidentally that could also be a disadvantage because of lack of cache data locality if a different thread now picks up that connection immediately (thus needing to fetch the data structures for this connection and flush the previous thread's cache). What works best will sensitively depend on your exact usage pattern.

If you are trying to process messages received subsequently on the same connection in different threads then this scheme to use epoll is not going to be appropriate for you, and an approach using a listening thread feeding an efficient queue feeding worker threads might be better.

Excellent ! Eventhough I am not confortable with the idea that this will block the write event from being processed in // with read event. — Apr 7 '11 at 20:11
It is possible to process read and write in parallel by using the dup() syscall to duplicate the fd and then registering one fd with EPOLLIN and the other with EPOLLOUT. However you have now used twice the number of fds and for applications that are ment to process many connections they tend to be a limited resource. — Apr 11 '11 at 19:54

Al Riddoch 48828 · Answer 3 · 2011-04-04 20:51:46Z

Previous answers that point out that calling epoll_wait() from multiple threads is a bad idea are almost certainly right, but I was intrigued enough by the question to try and work out what does happen when it is called from multiple threads on the same handle, waiting for the same socket. I wrote the following test code:

#include <netinet/in.h>#include <pthread.h>#include <stdio.h>#include <stdlib.h>#include <string.h>#include <sys/epoll.h>#include <sys/socket.h>#include <sys/types.h>#include <unistd.h>struct thread_info {  int number;  int socket;  int epoll;};void * thread(struct thread_info * arg){    struct epoll_event events[10];    int s;    char buf[512];    sleep(5 * arg->number);    printf("Thread %d start\n", arg->number);    do {        s = epoll_wait(arg->epoll, events, 10, -1);        if (s < 0) {            perror("wait");            exit(1);        } else if (s == 0) {            printf("Thread %d No data\n", arg->number);            exit(1);        }        if (recv(arg->socket, buf, 512, 0) <= 0) {            perror("recv");            exit(1);        }        printf("Thread %d got data\n", arg->number);    } while (s == 1);    printf("Thread %d end\n", arg->number);    return 0;}int main(){    pthread_attr_t attr;    pthread_t threads[2];    struct thread_info thread_data[2];    int s;    int listener, client, epollfd;    struct sockaddr_in listen_address;    struct sockaddr_storage client_address;    socklen_t client_address_len;    struct epoll_event ev;    listener = socket(AF_INET, SOCK_STREAM, 0);    if (listener < 0) {        perror("socket");        exit(1);    }    memset(&listen_address, 0, sizeof(struct sockaddr_in));    listen_address.sin_family = AF_INET;    listen_address.sin_addr.s_addr = INADDR_ANY;    listen_address.sin_port = htons(6799);    s = bind(listener,             (struct sockaddr*)&listen_address,             sizeof(listen_address));    if (s != 0) {        perror("bind");        exit(1);    }    s = listen(listener, 1);    if (s != 0) {        perror("listen");        exit(1);    }    client_address_len = sizeof(client_address);    client = accept(listener,                    (struct sockaddr*)&client_address,                    &client_address_len);    epollfd = epoll_create(10);    if (epollfd == -1) {        perror("epoll_create");        exit(1);    }    ev.events = EPOLLIN;    ev.data.fd = client;    if (epoll_ctl(epollfd, EPOLL_CTL_ADD, client, &ev) == -1) {        perror("epoll_ctl: listen_sock");        exit(1);    }    thread_data[0].number = 0;    thread_data[1].number = 1;    thread_data[0].socket = client;    thread_data[1].socket = client;    thread_data[0].epoll = epollfd;    thread_data[1].epoll = epollfd;    s = pthread_attr_init(&attr);    if (s != 0) {        perror("pthread_attr_init");        exit(1);    }    s = pthread_create(&threads[0],                       &attr,                       (void*(*)(void*))&thread,                       &thread_data[0]);    if (s != 0) {        perror("pthread_create");        exit(1);    }    s = pthread_create(&threads[1],                       &attr,                       (void*(*)(void*))&thread,                       &thread_data[1]);    if (s != 0) {        perror("pthread_create");        exit(1);    }    pthread_join(threads[0], 0);    pthread_join(threads[1], 0);    return 0;}

When data arrives, and both threads are waiting on epoll_wait(), only one will return, but as subsequent data arrives, the thread that wakes up to handle the data is effectively random between the two threads. I wasn't able to to find a way to affect which thread was woken.

It seems likely that a single thread calling epoll_wait makes most sense, with events passed to worker threads to pump the IO.

In my case I want to spawn multiple threads. Previous ideas suggesting to use one thread that in turn pumps data to another queue are not suitable for me for multiple reasons. — Apr 7 '11 at 20:28

Zan Lynx 28.9k34890 · Answer 4 · 2011-04-04 16:50:54Z

I believe that the high performance software that uses epoll and a thread per core creates multiple epoll handles that each handle a subset of all the connections. In this way the work is divided but the problem you describe is avoided.

Ok the problem is avoided, but inefficiency will arise, if there is a big difference in activity between different sets of sockets being managed by different threads. — Apr 7 '11 at 20:21

Jonathan 8,12631926 · Answer 5 · 2011-04-04 16:47:51Z

Generally, epoll is used when you have a single thread listening for data on a single asynchronous source. To avoid busy-waiting (manually polling), you use epoll to let you know when data is ready (much like select does).

It is not standard practice to have multiple threads reading from a single data source, and I, at least, would consider it bad practice.

If you want to use multiple threads, but you only have one input source, then designate one of the threads to listen and queue the data so the other threads can read individual pieces from the queue.

Why is it not a standard practise ? I've seen many project spawning multiple threads that do that call. Does epoll not behave like a Multiple Producers Multiple Consumers queue ? — Apr 7 '11 at 20:19
As I've seen it used, it was for handling asynchronous access to multiple streams by a single thread. I don't have much experience with it, though, so it is entirely possible that what I saw was uncommon use; if so, I apologize for my ignorance. — Apr 8 '11 at 13:27

Answer 6

PS ：第一个回答正好符合我自己的设计将send和read分在不同的线程来处理，可以借鉴一下，这里的方法是send线程一个epoll_wait 只关注fd的EPOLLOUT，read线程一个epoll_wait 只关注EPOLLIN 两个epoll_wait是不同的efd;

Answer 7

PS2: 自己的另外一个构思，send和read也放在不同的线程，但是只在read线程epoll_wait, send线程在缓冲满的时候将EPOLLOUT用epoll_ctl加入到read线程的efd中，read线程触发的EPOLLOUT只处理发送标志sending,send线程利用sending标志来判断是否可以发送数据，当然这里要对sending标志的处理要加锁，这个构思可以测试一下.这个构思源自另外一个网站的回答：

I apologize if I should not post this on LKML, but there seems to be
some lack of documentation for using epoll/AIO with threads. Are
these interfaces thread-safe? Can I use them safely in the following
way:

They are thread safe.

Thread A: while(1) { io_getevents(); ... }
// wait forever until an event occurs, then handles the event and loop

Thread B: while(1) { epoll_wait(); ... }
// same as thread A

Thread C: ... io_submit(); ...

Thread D: ... epoll_ctl(); ....

Suppose thread B calls epoll_wait and blocks before thread D calls
epoll_ctl. Is it safe to do so? Will thread B be notified for the
event submitted by thread D? Thread A and C pose the same question
for AIO.

Using the interfaces this way is pretty much their entire point. They'd be
almost useless if you couldn't use them in this way.

Answer 8

这两个构思都需要对socket的close进行特殊处理！

How to correctly read data when using epoll_wait

5 Answers