Memory Buffers

Introduction
Networking protocols place many demands on the memeory management facilities of the kernel. These demands include easily manipulating buffers of varying sizes, prepending and appending data to the buffers as the lower layers encapsulate data from higher layers, removing data from buffers(as headers are removed as data packets are passed up the protocol stack), and minimizing the amount of data copied for all these operations. The performance of the networking protocols is directly related to the memory management scheme used within in kernel.
In chapter 1 we look in more detail at mbufs and at the functions within the kernel that are used to manipulate them, as we will encounter mbufs on almost every page of the text. Understanding mbufs is essential for understanding the rest of the text.
The main use of mbufs is to hold the user data that travels from the process to the network interface, and vice versa. But mbufs are also used to contain a variety of other miscellaneous data: source and destination addresses, socket options, and so on.
Figure 2.1 shows the four different kinds of mbufs that we’ll encounter, depending on the M_PKTHDR and M_EXT flags in the m_flags member. The differences between the four mbufs in figure 2.1, from left to right, are as follows:

- If m_flags equals 0, the mbuf contains only data. These is room in the mbuf for up to 108 bytes of data(the m_dat array). The m_data pointer points somewhere in this 108-byte buffer. We show it pointing to the start of the buffer, but it can point anywhere in the buffer. The m_len member specifies the number of bytes of data, starting at m_data. Figure 1.6 was an example of this type of mbuf.
In figure 2.1 there are six members in the m_hdr structure, and its total size is 20 bytes. When we look at the c definition of this structure(figure 2.8)we’ll see that the first four members occupy 4 bytes each and the last two occupy 2 bytes each. We don’t try to differentiate between the 4-byte members and the 2-byte members in figure 2.1.
- The second type of mbuf has an m_flags value of M_PKTHDR, specifying a packet header, that is, the first mbuf describing a packet of data. The data is still contained within the mbuf itself, but because of the 8 bytes taken by the packet header, only 100 bytes of data fit within this mbuf(in the m_pktdat array). Figure 1.10 was an example of this type of mbuf.
The m_pkthdr.len value is the total length of all the data in the chain mbuf for this packet: the sum of the m_len values for all the mbufs linked through the m_next pointer, as shown in figure 1.8. The m_pkthdr.rcvif member is not used for output packets, but for received packets contains a pointer to the received interface’s ifnet structure(figure 3.6).
- The next type of mbuf does not contain a packet header(M_PKTHDR is not set) but contains more than 208 bytes of data, so an external buffer called a cluster is used(M_EXT is set). Room is still allocated in the mbuf itself for the packet header structure, but it is unused-we show it shaded in figure 2.1. Instead of using multiple mbufs to contain the data(the first with 100 bytes of data, and all the rest with 108 bytes of data each), Net/3 allocates a cluster of size 1024 or 2048 bytes. The m_data pointer in the mbuf points somewhere inside this cluster.
The Net/3 release supports seven different architectures. Four define the size of a cluster as 1024 bytes and three define it as 2048. The reason 1024 has been used historically is to save memory: if the cluster size is 2048, about on-quater of each cluster is unused for ethernet packet(1500 bytes maximum). We will see in section 27.5 that the Net/3 tcp never sends more than the cluster size per tcp segment, so with a cluster size of 1024, almost one-third of each 1500-byte ethernet frame is unused.
- The final type of mbuf contains a packet header and contains more than 208 bytes of data. Both M_PKTHDR and M_EXT are set.
There are numberous additional points we need to make about figure 2.1
- The size of the mbuf structure is always 128 bytes. This means the amout of unused space following the m_ext structure in the two mbufs on the right in figure 2.1 is 88 bytes(128-20-8-12).
- A data buffer with an m_len of 0 bytes is ok since some protocols(e.g., UDP) allow 0 length records.
- In each of the mbufs we show the m_data member pointing to the beginning of the corresponding buffer(either the mbuf buffer itself or a cluster). This pointer can point anywhere in the corresponding buffer, not necessarily the front.
- Mbufs with a cluster always contain the starting address of the buffer(m_ext.ext_buf) and its size(m_ext.ext_size). We assume a size of 2048 throughout this text. The m_data and m_ext.ext_buf members are not the same(as we show) unless m_data also points to the first byte of the buffer. The third member of the m_ext structure, ext_free, is not currently used by Net/3.
- The m_nextpkt pointer links multiple packets(records) together to form a queue of mbufs. Each packet on the queue can be a single mbuf or an mbuf chain. The first mbuf of each packet contains a packet header. If multiple mbufs define a packet, the m_nextpkt member of the first mbuf is the only one used-the m_nxetpkt member of the remaining mbufs on the chain are all null pointers.
Figure 2.2 shows an example of two packets on a queue. It is a modification of figure 1.8. We have placed the udp datagram onto the interface output queue(showing that the 14-byte ethernet header has been prepended to the ip header in the first mbuf on the chain) and have added a second packet to the queue: a tcp segment containing 1460 bytes of user data. The tcp data is contained in a cluster and an mbuf has been prepended to contain its ethernet, ip and tcp headers. With the cluster we show that the queue has a head pointer and a tail pointer This is how the interface output queues are handled in Net/3. We have also added the m_ext structure to the mbuf with the M_EXT flag set and have shaded in the unused pkthdr structure of this mbuf.

The first mbuf with the packet header for the udp datagram has a type of MT_DATA, but the first mbuf with the packet header for the tcp segment has a type of MT_HEADER. This is a side effect of the different way udp and tcp prepend the headers to their data, and makes no difference. Mbufs of these two types are essentially the same It is the m_flags value of M_PKTHDR in the first mbuf on the chain that indates a packet header.