epoll usage questions

From dw(http://www.developerweb.net), i saw the following questions about epoll, they are really good questions, please take some time to dig into them.

 

(1): ryujin

I have read the in Linux epoll is the way to go for scalable concurrentserver implementation and I have succesfully implemented a sampleprogram butI still have some small questions on epoll usage I hopesomeone here can answer.

1. Questions about epoll_wait():

1.1 Does maxevents parameter determine the maximun
number of concurrent clients?? If I want to handle
200,000 clients at any given time do I need to set this
variable to 200000??

1.2 Do I have to allocate the memory area pointed by
the *events argument. I tried with and without
allocating it and so far it works with both cases
but I have no definitive answer to this. If I do have
to allocate it do I need to allocate enough memory
to store maxevents*sizeof(struct epoll_event)??
In my case with 200000 concurrent clients would
this be overkill??

2. Question about timeouts

I can set a timeout in epoll_wait but I can not find any use
for this. I register 200,000 client sockets to epoll file descriptor
but I cannot timeout each socket individually. And epoll_wait
returns zero so there is no list of timed out sockets I can use
to remove timed out sockets.

2.1 How exactly should the epoll_wait timeout parameter be used?

2.2 How do I close sockets that have been idle for some time?

any answers/tips/resources are welcome...

Horacio

 

(2): i3839

1.1 No, it doesn't matter what you give as maxevents, it's just a hint.

It was used to determine the initial hashtable size in the kernel's epoll
implementation, but newer versions use a RB-tree, so it's ignored.

1.2 Respectively yes and no. The 'maxevents' given to epoll_wait is a different
one, it says how many struct epoll_events you have allocated in 'events'. For
this reason epoll_create was changed, it uses 'size' now, instead of 'maxevents.

If it works without allocating the memory you're unlucky and trashing some
random memory.

If 20 events are outstanding, but you handle ten events at a time, you'll allocate
struct epoll_events[10] and give 10 as maxevents. The first epoll_wait call will
return the first ten events, the next epoll_wait call will return immediately, giving
you the remaining events. The only reason to use a maxevents of more than 1 is
for efficiency, in case you get many events you can process them in batches
instead of one at a time, effectively eliminating epoll_wait's overhead.

Make your code work with maxevents = 1 first, finish the rest of your code,
and when everything really works and you want to optimise it, adjust the code
so that you can handle more event per epoll_wait call.

2.1 Without a timeout it would be hard to break out of an epoll_wait call.
Use it to limit how long epoll_wait blocks.

2.2 In general you don't close sockets when they're idle for a while, as most
sockets are idle most of the time in general. You've probably a very scalable
server, so those idle sockets aren't bothering you. Just let the clients close the
connection whenever they want.

If you've some protocol that wants to detect lost connections sooner and closes
the sockets of clients that don't respond in time, then you need to keep track
which sockets can timeout and for how long.

There are (at least) two ways to do this, use per socket timers or mark and sweep.

Timer method:

Basically you need a timer system that can set timers and handle triggered ones.
You can use existing timer systems like posix timers, see timer_create(2), or
something else. But those are probably not very scalable and generate a signal
when a timer expires, which is a bit icky. Better to make your own timer system,
and when you do the timeout parameter in epoll_wait is all you need.

The idea is that you set a timer for each socket you want to idle detect. If the
timer expires, you detected idleness. If you receive data from a socket, you reset
the timer.

Mark and sweep method:

Just keep track when the last time was you received data for each socket.
Then once in a while check all socket times and get rid of the idle ones.

See also the epoll(7) manpage.

Good luck.

 

(3): RobSeace

Also, on the idle sockets concept, let me suggest you setsockopt(SO_KEEPALIVE),
and since you're on Linux you can also set per-socket keep-alive timeouts via
TCP_KEEP{CNT,IDLE,INTVL} sockopts... (See "man 7 socket" and "man 7 tcp"...)
This way, the kernel will keep traffic flowing on idle connections, allowing you to
quickly detect connections that have silently gone dead somehow...

Well, SO_KEEPALIVE is pretty much universal, and I'd expect every system with any
kind of sockets implementation to support it... However, what is NOT universal is the
per-socket keep-alive timeouts... On most systems, SO_KEEPALIVE's timeouts are
fixed or only configurable on a system-wide basis where everything is affected...
Which tends to make it a lot less useful... For one thing, the standard default timeout
is outrageously long (generally 2 hours of idle time before it even starts to probe)...
And, most apps will generally want to detect problems FAR sooner than that... So, if
your system supports any kind of configurable keep-alive timeouts (preferably
per-socket like Linux, but even system-wide is better than nothing), I highly recommend
setting them to something saner than the defaults...

 

Reference:

(1): epoll usage questions:

http://www.developerweb.net/forum/showthread.php?t=5702

(2): Observing connection status on server site:

http://www.developerweb.net/forum/archive/index.php/t-4479.html

(3): Detecting timeout of idle connections

http://www.developerweb.net/forum/showthread.php?t=6187