Multiplayer and Network Programming (zz)

http://www.gamedev.net/community/forums/showfaq.asp?forum_id=15

We're very lucky, because this forum is one of the most on-topic forums, and one of the highest signal-to-noise forums on all of gamedev.net. Let's keep it this way! For example, if you ask something, but get no replies, please don't bump your thread. If you get no answer until the thread falls to the bottom of the first page, more bumps won't help. Re-formulating the question might, but please don't re-post the same question unless it's a very different formulation showing you've come to a deeper understanding of the problem since last time.

Some questions, while they are on-topic, do tend to come up again and again. I'll just give you the answers, and you don't have to go through all the trouble tpost and have to wait for an answer -- they're right here!

Q0) I have this dream to start the most uber MMORPG evar but I don't know whether I can write the networking part in javascript. Would Visual Basic be better?
Q1) I want to start learning about networking. Where do I go?
Q2) Should I use DirectPlay or WinSock?
Q3) Should I use TCP or UDP?
Q4) If TCP makes sure the data gets to me without me needing to worry about it, why would I want to use UDP, which makes no such guarantees?
Q5) Is UDP faster than TCP?
Q6) I'm sending data with TCP, but my ping times are insanely high! It seems the data is slowed down somwhere on the way. What's wrong?
Q7) Is there an easy wrapper on top of UDP which will give me reliable packets for packets that need it?
Q8) What are some other networking libraries that often get mentioned in the forum?
Q9) Why can't I host a game when I'm behind a firewall or DSL router?
Q10) Should I spawn a thread per connection in my game code?
Q11) Should I use blocking sockets, non-blocking sockets, or asynchronous sockets?
Q12) What should I put in my network packets, and how often should I send them?
Q13) Why can't my FD_SET be bigger than 64 on Windows, and 1024 on UNIX?
Q14) I'm trying to send two packets over TCP, but it seems like they arrive "merged" or one packet disappears on the receiving end. What's going on?
Q15) How do I send a struct using send() or sendto()?
Q16) How do I interpolate my character based on server lag, and still keep it in sync with the server?
Q17) Is a peer-to-peer MMORPG a good idea? Wouldn't that save a lot on server bandwidth?
Q18) How do I get the IP address of the machine I'm running on?
Q19) No, really. I want to use TCP for my MMORPG.
Q20) I need a server that will stay up longer than my cable modem, but typical web hosts won't let me run my own server program. What are my options?

---

Q0) I have this dream to start the most uber MMORPG evar but I don't know whether I can write the networking part in javascript. Would Visual Basic be better?
A0) The best place to start is here.
After you have assimilated the full knowledge of that highly relevant article, you're ready to move on to answer 1 below, or a link such as this.

---

Q1) I want to start learning about networking. Where do I go?
A1) Try Beej's Networking Tutorials. You could also try the intro section right here on this site.
Also try the WinSock FAQ for sockets programming, especially as relates to Windows.

---

Q2) Should I use DirectPlay or WinSock?
A2) As of Summer Update 2004, Microsoft has put DirectPlay into "sunset" mode, where they will not develop the API further. Microsoft now recommends you use WinSock for networking, until the re-cast X-box Live! technology is rolled out for PCs, probably around Longhorn time-frame. If you're on Linux, MacOS, or some other platform like that, life is easier: you only have regular sockets to work with. Luckily, sockets and WinSock code can look very similar if you stick to the Berkeley sockets functions of WinSock, and avoid WSA functions; you can write quite portable programs with a little bit of thought.

Remember to include <winsock2.h> before including <windows.h> in your win32 program, or compile errors will ensue. Also, always start your WinSock program with WSAStartup( MAKEWORD(2,2), ... ) to get version 2.2 of WinSock, as the older versions have bugs and limitations that you don't want to deal with. Finally, you need to link with ws2_32.lib on Win32 when using WinSock.

---

Q3) Should I use TCP or UDP?
A3) There are at least four different kinds of games as far as networking is concerned: Turn-based games, Real Time Strategy games, Role Playing Games, and Action Games. Briefly, if you're turn-based, you should go with TCP because it's simpler, unless you need peer-to-peer NAT punch-through. For RTS games, you're usually best off using TCP, although UDP can make sense in extreme cases. For role playing games, the story is less clear -- action-based RPGs with lots of kinetics, like City of Heroes, use UDP, whereas slower RPGs and MUDs often stay with TCP. For action-based games like first-person shooters or racers, you should absolutely use UDP.

---

Q4) If TCP makes sure the data gets to me without me needing to worry about it, why would I want to use UDP, which makes no such guarantees?
A4) TCP is a stream-based protocol. What goes in one end, will come out the other end, in order, with no duplication or packet drops. This is very convenient when correctness is paramount. However, to make this guarantee, TCP must suspend all packet delivery to the receiving end, if one packet is lost, until that packet can be detected as lost and re-sent. This may cause hick-ups of several seconds for a single packet that's lost!

---

Q5) Is UDP faster than TCP?
A5) UDP uses slightly smaller headers per packet than TCP, which may give you marginally better throughput on narrow channels (such as modems). UDP will also deliver any received packet as it is received, without waiting to re-order packets, so if latency is more important than correctness of the underlying channel, then UDP will introduce less jitter in your game, and gameplay will be smoother.

---

Q6) I'm sending data with TCP, but my ping times are insanely high! It seems the data is slowed down somwhere on the way. What's wrong?
A6) When you send data on a TCP connection, the network packet typically implements something called Nagle's algorithm. This will make the system not immediately send data if you don't fill up an MTU's worth of data in a single write; instead it will sit and wait for some time (200 milliseconds is typical), hoping that you will write more data that it can coalesce into a single packet. You turn OFF this delay by turning ON the TCP_NODELAY socket option.

---

Q7) Is there an easy wrapper on top of UDP which will give me reliable packets for packets that need it?
A7) I hear Enet is pretty good, although the documentation is pretty sparse.

---

Q8) What are some other networking libraries that often get mentioned in the forum?
A8) Low-level "sockets portability" libraries include: Enet, HawkNL, PLib/Medusa and SDL_net.

High-level "we give you structure" libraries include: ClanLib, Net-Z/Eterna, Nevrax/NEL, OpenSkies, OpenTNL, RakNet and ReplicaNet.
Mention (or the absense of mention) of a specific library does not mean gamedev.net or the moderator endorses (or counter-endorses) that library.

---

Q9) Why can't I host a game when I'm behind a firewall or DSL router?
A9) You're probably suffering from the Network Address Translation in the firewall/router. There's a number of documents describing how to set up an introducer system to punch through NAT and allow you to host from behind a firewall. Try this web page or wait for my upcoming article in Game Programming Gems 5 (out spring 2005).

---

Q10) Should I spawn a thread per connection in my game code?
A10) The short answer is "likely no." The longer answer is that, if you're using UDP, you should only need a single socket, from which you can read a packet, process a packet, repeat. If you're using TCP, you will need a socket per player, but the players are likely sharing and affecting a single world state inside the program, which needs to be protected by locking; thus, the extra threads would just end up blocking on locks anyway, so using select() and reading out what's there, then processing it, is recommended. Some users find "co-operative threads" or fibers to be a nice middle ground, because it gives some of the advantages of real threads (a stack per context) without the locking overhead (because yield is explicit).

---

Q11) Should I use blocking sockets, non-blocking sockets, or asynchronous sockets?
A11) On UNIX (at least Linux), asynchronous sockets are not generally available, although you can arrange to get a signal when "any fd" is ready (FIOASYNC). If you're on windows, and you're prepared to deal with the significant management overhead, they can lead to good performance. However, using non-blocking sockets in concordance with select() is usually not only the easiest to implement correctly, but will also perform well -- I recommend not doing anything fancier unless a profiler tells you to (i e, until oprofile tells you you're spending 10% of your CPU time inside select()).

Blocking sockets makes it much harder to write a real-time game, because, well, they block! If you're writing a file transfer program or something like that, especially if it's point-to-point (only a single connection), blocking TCP sockets are very easy to work with, assuming you can live within the constraints of that model.

If you are Windows-only, a high-performance alternative is Overlapped I/O on sockets.

Last, there's a good write-up of TCP/socket performance, with some comparisions of Unix and Windows (although mostly Unix-based), that's recommended for people worried about throughput.

---

Q12) What should I put in my network packets, and how often should I send them?
A12) That depends on your goals and your game.
For an FPS style game, you might want to read about how Quake III, Unreal and Tribes does it. (If that link doesn't work, try webarchive.org, although that's very slow.
For a turn-based game, the most important thing is to make sure that players stay in sync; bandwidth and latency doesn't play into it as much.
For a real-time strategy game, bandwidth compression becomes paramount, because there is often a large number of units moving around. A good article is on GamaSutra.com.
For data compression approaches, Jon Blow wrote a series in Game Developer: here.
There's also this thread on the forum about RPG and RTS command/packet encoding that might be helpful.
For MMORPGs, the typical solution is to send the position & state of some entity (possibly with delta compression and periodic baselines), every so often, where you send it more often the closer the player is to the entity. If you have a good link describing this, please let me know!

---

Q13) Why can't my FD_SET be bigger than 64 on Windows, and 1024 on UNIX?
The implementation of FD_SET on Windows is a big array of socket values, and adding, testing, and removing sockets is linear in number of sockets in the set. Luckily, modern caches make this not be as terrible as you might first think, but hard-coding a solution that loops over the FD_SET specifically to extract ready sockets is probably a good idea. Windows does this because its sockets are not low integer file descriptors.
The implementation on UNIX is a bit mask, where you set bit N if you want to examine file descriptor N.
Both of these implementations can easily be extended if you declare your own file set structure compatible with the operating system structure, but bigger. Most system headers will also allow you to define FD_SETSIZE on the command line, to make for a bigger set size without re-declaring the structure.

---

Q14) I'm trying to send two packets over TCP, but it seems like they arrive "merged" or one packet disappears on the receiving end. What's going on?
TCP is a stream protocol. Everything you push in one end will arrive, in order, out the other end. However, the "boundaries" you establish by calling send() on one end are not preserved in the stream. The TCP implementation may split the data such that multiple send() calls get merged into one recv() call, or so that a single, large send() gets received through multiple recv() calls.
To construct message semantics on TCP, each time you send a message, you have to precede it with the length of the message, as a short or a long (use htons() or htonl() to make it cross-platform). Then write that many bytes of message. On the receiving side, you'd first read out the length (a short or long) and then that many bytes of message data, re-constituting each message sent from the other end.

---

Q15) How do I send a struct using send() or sendto()?
The socket send functions may be prototyped using char * in some older headers, but they really take just a byte pointer and a size. You send a struct the same way you would write it to a file. To send and receive a player position update, some simple code might look something like this:

  struct PlayerPosPacket {    unsigned short code;    unsigned short player;    float pos_x, pos_y, pos_z;    float vel_x, vel_y, vel_z;    float heading;  };  PlayerPosPacket packet;  packet.code = PLAYER_POS;  packet.player = thePlayerId;  ...  int r = sendto( sock, (char const *)&packet, sizeof(packet), 0, (sockaddr *)&dst_addr, sizeof(dst_addr) );  if( r < 0 ) { error(sock); }  union {    PlayerPosPacket playerPos;    SomeOtherKindOfPacket someOther;  } recvPacket;  int r = recvfrom( sock, (char *)&recvPacket, sizeof( recvPacket ), 0, (sockaddr *)&src_addr, sizeof(src_addr) );  if( r < 0 ) { error(sock); }  /* assume all packet types start with unsigned short code; */  switch( recvPacket.playerPos.code ) {    case PLAYER_POS:      if( r != sizeof(recvPacket.playerPos) ) { formatError(); }      do_playerPos( recvPacket.playerPos, src_addr );      break;    case SOME_OTHER:      if( r != sizeof(recvPacket.someOther) ) { formatError(); }      do_someOther( recvPacket.someOther, src_addr );      break;    ...  }

Note that, for brevity, this code does not make the code byte order agnosting using htonl()/htons().

If you want to send more complicated objects (classes with vtables, data structures that use pointers, etc), then you need to serialize your object to a sequential array of bytes before sending, and then de-serialize on receipt. Serialization is typically covered in advanced tutorials for the programming language of your chocie.

---

Q16) How do I interpolate my character based on server lag, and still keep it in sync with the server?
There's a great description in the Zen of Networked Character Physics thread.

---

Q17) Is a peer-to-peer MMORPG a good idea? Wouldn't that save a lot on server bandwidth?
First, server bandwidth isn't actually the first, or even second cost for an operating MMORPG anymore. Content production, customer service, and other people-related costs are usually much higher.

Second, the two main technical challenges of peer-to-peer MMORPGs are hard to solve in a sufficiently satisfactory manner for current game designs:
- How to secure the game state data such that a user could not hack his client, or data files, or attach to the client with a machine monitor when running, and change the simulation to his advantage. This is, technically, not possible without a secure-computing platform like the much-maligned Palladium.
- How to find the right other peers to interact with with sufficiently low latency, and update this mapping when players come and go.

For now, peer-to-peer works best for transfer of large bulk items where you can easily verify correctness, such as content downloads and patches. If you're thinking about posting a thread regarding your great new peer-to-peer MMORPG design, make SURE you've written some code, and actually tested it on at least three collaborating machines, before you start. Else most people will likely just sigh at your unfettered newbie optimism :-)

There's some existing published research, for example that collected into the VAST project. You should check up on this, too, if you're interested in P2P MMOGs.

---

Q18) How do I get the IP address of the machine I'm running on?
Typically, it's a bad idea to get your own IP address, because it's not certain that the address is visible to people outside your router. Also, many machines have more than one IP address (each interface has one in a machine). Last, not all ISPs or routers have DNS/reverse DNS set up such that gethostname() followed by gethostbyname() will give you the right answer.

To use command-line tools to list your addresses on Windows, use "ipconfig". On Linux, use "ifconfig". On BSD, use "ifconfig" but you probably have to be root.

To programmatically determine the CORRECT address to use, you have to bounce a query off a server that's available on the public internet, and look at what address it returns back. This will work correctly for situations where you're behind NAT (routers/firewalls).

If you want to know the list of all interface addresses, say for selectively binding to only some interface, then you need to use OS-specific system calls; i e you need to do whatever ipconfig/ifconfig does, which involves the network device API on Windows, and walking /proc on Linux.

---

Q19) No, really. I want to use TCP for my MMORPG.
Sure, it can be done. Check out this long-running and productive thread for a large variety of viewpoints and suggestions.

---

Q20) I need a server that will stay up longer than my cable modem, but typical web hosts won't let me run my own server program. What are my options?
Assuming you don't have friends or a school that'll allow you to run your internet server process, you have three basic options:
1) Self-managed Hosting
2) Server Co-location
3) Roll Your Own

1) Self-managed hosting is like web hosting, except you actually lease a full machine, with some amount of network bandwidth included. You get root access to the machine (or Administrator, for Windows), and can install whatever you want. You typically get access to some web page that will allow you to reboot your machine if necessary. If you screw it up, you have to pay the hosting company to re-image the machine (which will lose your files, so have back-ups!). Places include ServerBeach, 1-and-1, ServerMatrix, and a host of others. Do some research on their customer service and availability, because it varies wildly. Prices start at around $69/month, with everything from 200 GB to 2000 GB network transfer per month included. Watch out for overage charges!

2) Server Co-location is the next step up. You buy the hardware, and lease some space in a rack or cage at a well-connected facility. You get physical access to the facility, and they provide you with what's basically an Ethernet cable wired to the Internet. You agree on some level of service an bandwidth, which will determine the price. Typically, traffic is measured over 5 minute intervals, and you'll pay for the amount of traffic used by the 95th percentile measurement each month. Prices start out higher than self-managed plans, just because of the physical set-up, but when you have ten servers or more, co-location starts looking pretty attractive (assuming you live within reach of a co-lo hosting center, of course). Typical costs for a set of servers runs four digits per month.

3) Roll Your Own: Buy the hardware. Buy the management and control infrastructure. Buy the cooling. Get a wider power cable from your utility company. Fight city zoning ordinances. Take out a mortgage on your house to afford the up-front cost of DS-3 service and up (45 Mbps). Find an internet service provider that will agree to route your traffic. Congratulations! You just spent months building your own data center, and you're now in business! Expect to pay five digits to build a sufficiently heavy-duty set-up.