Possible Firewall-Induced Network Problems

The following is an excerpt from Sendmail Performance Tuning by Nick Christenson, pp. 60-61, published by Addison-Wesley, Copyright 2003.

Different MTUs in the Network

The problem of networks having different maximum packet sizes on the Internet is handled in two ways. First, packets can be fragmented by routers as needed along the way. This is easy on the sending host, but requires a router to perform extra work. Second, one can determine what the maximum packet size that all networks between the two hosts can handle by sending large packets and receiving messages back from intermediaries that can't handle those large packets. This process, called "Path MTU Discovery", is documented in RFC 1191. The benefits of this method versus the alternatives are eloquently stated in the RFC. However, every packet that is sent to a remote host that is larger than the maximal MTU of an intermediate network must be be re-sent at least once. This extra overhead might be avoided by setting the network interface's MTU to a ... modest size, ... .

A big downside associated with Path MTU Discovery bears mentioning. Servers using this protocol typically send IP packets with the "don't fragment" bit set. Then, when a router cannot deliver the packet to the next hop, it will return an ICMP "Destination Unreachable" message to indicate that the packet cannot be processed. RFC 1191 specifies that the MTU for the next network hop be encoded in this ICMP packet. That way, the originating server will know the size the packet should be when it is re-sent, and it can iteratively determine the largest packet size that can be sent without fragmentation.

A problem occurs when a router between the two communicating hosts blindly discards ICMP packets. In this case, the sending host doesn't know what to do. It never receives an acknowlegement for the packet it sent, yet no ICMP Destination Unreachable packet is returned to inform it that something went wrong. A common manifestation of this problem is the successful transmission of very small email messages, while larger messages (perhaps totaling more than some "magic" size, like 536 or 1,500 bytes) can't get through.

This situation is not as uncommon as it ought to be. In the name of security concerns, a significant number of sites have configured packet filters on their routers or firewall boxes to reject all ICMP packets. While this cure might seem worse than the disease, and certainly I don't endorse this strategy, it's not surprising that some sites choose this course of action. Many fears concerning ICMP-based network attacks, especially distributed denial-of-service (DDoS) attacks, are justified. Path MTU Discovery is a relatively esoteric part of the IP protocol, and a large percentage of the people responsible for implementing network security policy at sites around the Internet have likely never heard of it. Consequently, they don't understand why they need to support it. This policy may be wrong, but in this day and age it's understandable.

If an organization is having problems getting email through to other sites, and Path MTU Discovery is the culprit, it's probably best to disable this feature on the affected servers. Of course, if a site's own network is causes the problem, then the obvious fix is to allow ICMP Destination Unreachable packets through to the email servers. While it turns out that offloading the effort involved in packet fragmentation may decrease the load on email servers, any benefit will be very slight, if it's measurable at all.