VDS Best Practices – Design Considerations (Part 1 of 6)

After a long break I am back again on this forum. Over the last couple of months I was busy interacting with various customers as part of the VMworld US and Europe conferences and trying to understand how customers are deploying distributed switch in their environment. I have also spent lot of time internally talking to VMware R&D engineers as well as Field experts (SEs and PSOs) on what guidance we would like to provide to our customers regarding the deployment of vSphere Distributed Switch (VDS). After all these discussions, I have come up with some guidelines or best practices that customers have to consider while deploying VDS. These guidelines or best practices are discussed with reference to a sample example deployment. It is important to note that the suggestions/guidelines discussed here are based on the assumptions of the example deployment. Every customer environment is different as well as customer needs are different, and customers have to take that into account while designing the virtual network infrastructure using VDS. VDS is a flexible platform that can support different requirements from different customers. 

So let’s first dive into the different design considerations that you should consider before you go and start deploying VDS in your environment.

Design Considerations

Three main aspects influence the design of a virtual network infrastructure:

1)   Customer’s infrastructure design goals

2)   Customer’s infrastructure component configurations

3)   Virtual infrastructure traffic requirements

Let’s take a look at each of these aspects in a little more detail.

Infrastructure design Goals

Customer’s want their network infrastructure Available 24/7, Secure from any attacks, and Perform efficiently throughout the day-to-day operations. In the case of a virtualized environment these requirements become much more demanding as more and more business critical applications run on consolidated environment. These requirements on the infrastructure translates into design decisions that should incorporate following best practices for virtual network infrastructure:

• Avoid any single point of failure in the network
• Isolate each traffic type
• Make use of traffic management and optimization capabilities

Infrastructure component configurations

In every customer environment, the compute and network infrastructure used differ in terms of configuration, capacity, and feature capabilities. These different infrastructure component configurations influence the virtual network infrastructure design decisions. The following are some of the configurations and features that administrators have to look out for.

• Server configuration: Rack or Blade servers
• Network adapter configuration: 1 Gigabit or 10 Gigabit Ethernet network adapters; number of available adapters; offload function on these adapters if any.
• Physical Network Switch Infrastructure capabilities: Switch clustering

It is impossible to cover all the different virtual network infrastructure design deployments based on the various combinations of type of servers, network adapters and network switch capability parameters. In this paper, the following four commonly used deployments are described that are based on standard rack server and blade server configurations:

• Rack Server with Eight 1 Gigabit Ethernet network adapters Deployment
• Rack Server with Two 10 Gigabit Ethernet network adapters
• Blade Server with Two 10 Gigabit Ethernet network adapters
• Blade Server with Hardware assisted Multiple Logical Ethernet network adapters

It is assumed that the network switch infrastructure has standard layer 2 switch features (High availability, redundant paths, fast convergence, port security) available to provide reliable, secure and scalable connectivity to the server infrastructure.

Virtual Infrastructure traffic

VMware vSphere virtual network infrastructure carries different traffic types. To mange the virtual infrastructure traffic effectively, vSphere and network administrators have to understand the different traffic types and their characteristics. The following are the key traffic types that flow in the VMware vSphere infrastructure along with their traffic characteristics:

• Management traffic: This traffic flows through a vmknic and carries ESXi host to vCenter configuration and management communication as well as ESXi host to ESXi host High Availability (HA) related communication. This traffic has low network utilization but has very high availability and security requirements.
• VMware vMotion traffic: With advancement in the vMotion technology, a single vMotion can consume almost a full 10 Gigabit bandwidth. A maximum of eight simultaneous vMotions can be performed on a 10 Gigabit uplink while on a 1 Gigabit uplink four simultaneous vMotions are allowed. vMotion traffic has very high network utilization and can be bursty at times. Customers have to make sure that vMotion traffic doesn’t impact other traffic types because it could consume all available I/O resources. Another property of vMotion traffic is that it is not sensitive to throttling and thus makes a very good candidate to perform traffic management on.
• Fault Tolerant traffic: When fault tolerance logging is enabled for a virtual machine, all the logging traffic is sent to the secondary fault-tolerant virtual machine over a designated vmknic port. This FT process could require a lot of bandwidth at low latency as it replicates the I/O traffic and memory state information to the secondary virtual machine.
• iSCSI/NFS traffic:  IP storage traffic is carried over vmknic ports and this traffic varies according to disk I/O requests. With end-to-end jumbo frame configuration, more data is transferred with each Ethernet frame reducing the number of frames on the network. This larger frame reduces the overhead on servers and targets and improves the IP storage performance. Congested and lower speed networks can cause latency issues that disrupt access to IP storage. It is recommended to provide high-speed path for IP storage and avoid any congestion in the network infrastructure.
• Virtual Machine Traffic: Depending on the workloads that are running on the guest virtual machine, the traffic patterns will vary from low to high network utilization. Some of the applications running in virtual machines might be latency sensitive as is the case with VOIP workloads.

 The following Table 1 summarizes the characteristics of each traffic type

 Table 1 Traffic Types and characteristics

Traffic Type

Bandwidth Usage

Other Traffic requirements

Management

Low

Highly reliable and Secure channel

vMotion

High

Isolated channel

Fault Tolerant

Medium to High

Highly reliable low latency channel

iSCSI

High

Reliable and high speed channel

Virtual Machine

Depends on Application

Depends on applications.

 

To understand the different traffic flows in the physical network infrastructure, network administrators use Network Traffic management tools. These network management tools help monitor the physical infrastructure traffic but do not provide visibility into virtual infrastructure traffic. With the release of vSphere 5, VDS now supports the NetFlow feature. The NetFlow feature allows exporting the internal (VM to VM) virtual infrastructure flow information to standard network management tools. Administrators now have the required visibility in to virtual infrastructure traffic. This helps administrators monitor the virtual network infrastructure traffic through a familiar set of network management tools. Customers should make use of the network data collected from these tools during the capacity planning or network design exercises.

 

Example Deployment components

After looking at the different design considerations, this section provides the list of components that are used in an example deployment. This example deployment is used to help illustrate some standard VDS design approaches. The following are some common components in the virtual infrastructure. The list doesn’t include storage components that are required to build the virtual infrastructure. It is assumed that customer will deploy IP storage in this example deployment.

Hosts

Four ESXi hosts providing compute, memory and network resources according to the configuration of the hardware. Customers can have different number of hosts in their environment based on their needs. One VDS can span across 350 hosts. This capability to support large number of hosts provides the required scalability to build a private or public cloud environment using VDS.

Clusters

A cluster is a collection of VMware ESXi hosts and associated virtual machines with shared resources. Customers can have as many clusters in their deployment as required. With one VDS spanning across 350 hosts, customers have the flexibility of deploying multiple clusters with different number of hosts in each cluster. For simple illustration purpose two clusters with two hosts each are considered in this example deployment. One cluster can have maximum 32 hosts.

vCenter Server

VMware vCenter server centrally manages a VMware vSphere environment. Customers can manage VDS through this centralized management tool. vCenter Server can be deployed on a virtual machine or a physical host. The vCenter Server is not shown in the diagrams but customers should assume that it is present in this example deployment. vCenter Server is only used to provision and manage VDS configuration. Once provisioned, hosts and virtual machines network operate independent of vCenter Server. All components required for network switching reside on ESXi hosts. Even if the vCenter Server fails, the hosts and virtual machines will still be able to communicate.

 In the deployment where vCenter Server is hosted on a virtual machine, customers have to pay more attention to the network configurations. In such deployments, if the networking for virtual machine hosting vCenter server is misconfigured, then the network connectivity of vCenter Server is lost. This mis-configuration must be fixed. However, you need vCenter Server to fix the network configuration because only vCenter Server can configure a VDS. As a workaround to this situation, customers have to reconnect the virtual machine hosting vCenter Server to a standard vSwitch using vSphere Client that is directly connected to the host. To do this, a VSS (vSphere standard switch) must be on a host that is also connected to the management network of hosts.

 

Network Infrastructure

Physical network switches in the access and aggregation layer provide connectivity between ESXi hosts and to the external world. These network infrastructure components support standard layer 2 protocols providing secure and reliable connectivity.

Along with the above four components of the physical infrastructure in this example deployment, some of the virtual infrastructure traffic types are also considered during the design. The following section describes the different traffic types in the example deployment.

Virtual Infrastructure Traffic Types

In this example deployment, there are standard infrastructure traffic types that include iSCSI, vMotion, FT, Management along with virtual machine traffic type. Customers may have other traffic types in their environment based on the choice of the storage infrastructure (FC,NFS,FCoE). The diagram in the Figure 1 below shows the different traffic types along with associated port groups on an ESXi host. It also shows the mapping of the network adapters to the different port groups.

Figure 1 Different Traffic types running on a Host

After covering the different design considerations and the example deployment, we will turn our attention to the different VDS design options around the example deployment. In the next blog entry I will cover the important virtual and physical switch parameters that customers should consider in all these design options.