Virtual Networking

The ability to manage virtual machines is something which is receivinga lot of focus right now. Xen, KVM, QEMU and others provide theinfrastructure required to run a virtual machine, and each can provideguests with a virtual network interface. This proposal addresses theproblem of how guests are networked together.

We aim:

• To make virtual networking "just work".

  Guests should be able to communicate with each other, their host and the Internet without any fuss or configuration. This should be the case even with laptops and offline machines.

• To allow a greater flexibily with how guests are networked.

  It should be possible to isolate groups of guests in different networks, allow guests on different physical machines to communicate, firewall guests' networks from physical networks or allow guests to appear just like physical machines on physical networks.

• To make networking virtual machines analogous with networking physical machines.

• To support inter-networking between virtualisation technologies.

User Visible Concepts

It's important to consider the manner in which we expose thefunctionality of virtual networking. What concepts will be exposingthrough the UI? Are those concepts well defined and consistent? Arethose concepts more complex than neccessary? Or are the too simple tobe able to support the functionality we want?

Real world, or "physical", concepts[1]:

• Network - a number of interconnected machines.
• Network Interface - hardware which enables a machine to connect to a network.
• Bridge - hardware which allows enables the interconnection of machines to form a network. Bridges can also be connected to other bridges to form a larger network.
• Router - hardware which connects two or more distinct networks, allowing machines on different networks to communicate with one another. Sometimes a router and a bridge are available as a combined piece of hardware - the bridge forms a network and the router connects that network to another distinct network.
• Firewall - software on a router which can be used to control how machines on an "external" network (e.g. the Internet) can communicate with machines on an "internal" network. For a given type of connection, you can choose to disallow connections of a that type or forward them to a specific internal machine. Can also be used to control how internal machines can communicate with external machines.

With virtual networking, we will be exposing the following "virtual"concepts:

• Virtual Network - a number of interconnected virtual machines.
• Virtual Network Interface - a network interface in a virtual machine.
• Virtual Bridge - allows the interconnection of virtual machines to form a virtual network. A virtual bridge may be configured to also act as a virtual router and firewall. A virtual bridge may also be connected to another virtual bridge (perhaps on another physical machine) to create a larger virtual network.

(Note, unprivileged users may create any of the above)

Finally, where the physical world meets the virtual world:

• Shared Physical Interface - if a physical interface is configured to be "shared", then any number of virtual interfaces may be connected to it allowing virtual machines to be connected to the same physical network which the physical interface is connected to.

  Only privileged users may configure a physical interface to be shared and/or connect guests to it.

There are a few problems with all of the above:

1. The distinction between a bridge and a router requires a lot of technical knowledge to fully understand. However, the model of e.g. a LinkSys router is familiar to a lot of people - a box which allows you to network your machines together and connect that network to (and firewall off) the Internet.
2. This "shared physical interface" notion is very "makey upey". We could perhaps talk about the idea in terms of connecting a physical interface to a virtual bridge, but it exposes the bridge vs. router distinction more than we'd like.
3. Guests are connected to a specific physical interface, whereas perhaps users wish guests to be connected to "the network" - i.e. if NetworkManager switched from wireless to wired while remaining on the same subnet, perhaps we'd like to automatically switch the bridge to the new network. In reality, though, bridged networking is only really sane for machines on a fairly static network connection.

[1] - Yes, these definitions aren't entirely accurate, but theydescribe the kind of understanding a moderately technical user mighthave of the concepts.

Example Networks

Below are some example networks users may configure and an explanationof how that network would be implemented in practice.

1. A privileged user creates two (Xen) guests, each with a Virtual Network Interface. Without any special networking configuration, these two guests are connected to a default Virtual Network which contains a combined Virtual Bridge/Router/Firewall.

     +-----------+                   D           +-----------+  |   Guest   |           N   D   H           |   Guest   |  |     A     |           A   N   C           |     B     |  |   +---+   |           T   S   P           |   +---+   |  |   |NIC|   |           ^   ^   ^           |   |NIC|   |  +---+-+-+---+           +---+---+           +---+-+-+---+        ^                     |                     ^        |   +--------+    +---+---+    +--------+   |        +-->+ vif1.0 +----+ vnbr0 +----+ vif2.0 +<--+            +--------+    +-------+    +--------+

   Notes:

   • "vnbr0" is a bridge device with it's own IP address on the same subnet as the guests.
   • IP forwarding is enabled in Dom0. Masquerading and DNAT is implemented using iptables.
   • We run a DHCP server and a DNS proxy in Dom0 (e.g. dnsmasq)

2. A privileged user does exactly the same thing as (1), but with QEMU guests.

                                     D                          N   D   H                          A   N   C                          T   S   P                          ^   ^   ^                          +---+---+                              |                          +---+---+  +-----------+           | vnbr0 |           +-----------+  |   Guest   |           +---+---+           |   Guest   |  |     A     |               |               |     B     |  |   +---+   |           +---+---+           |   +---+   |  |   |NIC|   |           | vtap0 |           |   |NIC|   |  +---+-+-+---+           +---+---+           +---+-+-+---+        ^       +-------+     |     +-------+       ^        |       |       | +---+---+ |       |       |        +------>+ VLAN0 +-+  VDE  +-+ VLAN0 +<------+                |       | +-------+ |       |                +-------+           +-------+

   Notes:

   • VDE is a userspace ethernet bridge implemented using vde_switch
   • "vtap0" is a TAP device created by vde_switch
   • Everything else is the same as (1)
   • This could be done without vde_switch by having Guest A create vtap0 and have Guest B connect directly to Guest A's VLAN. However, if Guest A is shut down, Guest B's network would go down.

3. An unprivileged user does exactly the same thing as (2).

     +-----------+                               +-----------+  |   Guest   |          +----+----+          |   Guest   |  |     A     |          |userspace|          |     B     |  |   +---+   |          | network |          |   +---+   |  |   |NIC|   |          |  stack  |          |   |NIC|   |  +---+-+-+---+          +----+----+          +---+-+-+---+        ^       +-------+     |     +-------+       ^        |       |       | +---+---+ |       |       |        +------>+ VLAN0 +-+  VDE  +-+ VLAN0 +<------+                |       | +-------+ |       |                +-------+           +-------+

   Notes:

   • Similar to (2) except there is can be no TAP device or bridge
   • The userspace network stack is implemented using slirpvde to provide a DHCP server and DNS proxy to the network, but also effectively a SNAT and DNAT router.
   • slirpvde implements ethernet, ip, tcp, udp, icmp, dhcp, tftp (etc.) in userspace. Completely crazy, but since the kernel apparently has no secure way to allow unprivileged users to leverage the kernel's network stack for this, then it must be done in userspace.

4. Same as (2), except the user also creates two Xen guests.

       +-----------+                   D           +-----------+   |   Guest   |           N   D   H           |   Guest   |   |     A     |           A   N   C           |     B     |   |   +---+   |           T   S   P           |   +---+   |   |   |NIC|   |           ^   ^   ^           |   |NIC|   |   +---+-+-+---+           +---+---+           +---+-+-+---+         ^                     |                     ^               |   +--------+    +---+---+    +--------+   |               +-->+ vif1.0 +----+ vnbr0 +----+ vif2.0 +<--+                   +--------+    +---+---+    +--------+                                         |                                                       +---+---+                                                   | vtap0 |                                                   +---+---+                                                       |                                           +-------+    +--+--+   +-------+                       +---->+ VLAN0 +----+ VDE +---+ VLAN0 +<-----+        |     +-------+    +-----+   +-------+      |        V                                           V   +---+-+-+---+                               +---+-+-+---+  |   |NIC|   |                               |   |NIC|   |  |   +---+   |                               |   +---+   |  |   Guest   |                               |   Guest   |  |     C     |                               |     D     |  +-----------+                               +-----------+

   Notes:

   • In this case we could do away with VDE and have each QEMU guest use its own TAP device.

5. Same as (3) except Guests A and C are connected to a Shared Physical Interface.

    +-----------+                 |          D       +-----------+ |   Guest   |          ^      |  N   D   H       |   Guest   | |     A     |          |      |  A   N   C       |     B     | |   +---+   |      +---+---+  |  T   S   P       |   +---+   | |   |NIC|   |      | eth0  |  |  ^   ^   ^       |   |NIC|   | +---+-+-+---+      +---+---+  |  +---+---+       +---+-+-+---+       ^                |      |      |                 ^       | +--------+ +---+---+  |  +---+---+ +--------+  |       +>+ vif1.0 +-+ ebr0  +  |  + vnbr0 +-+ vif2.0 +<-+         +--------+ +---+---+  |  +---+---+ +--------+                          |      |      |                    +---+---+  |  +---+---+                    | vtap1 |  |  | vtap0 |                    +---+---+  |  +---+---+                        |      |      |          +-------+  +--+--+   |   +--+--+  +-------+       +->+ VLAN0 +--+ VDE +   |   + VDE +--+ VLAN0 +<-+       |  +-------+  +-----+   |   +-----+  +-------+  |       V                       |                       V  +---+-+-+---+                 |                 +---+-+-+---+ |   |NIC|   |                 |                 |   |NIC|   | |   +---+   |                 |                 |   +---+   | |   Guest   |                 |                 |   Guest   | |     C     |                 |                 |     D     | +-----------+                 |                 +-----------+

   Notes:

   • The idea here is that when the admin configures eth0 to be shareable, eth0 is configured as an addressless NIC enslaved to a bridge which has the MAC address and IP address that eth0 should have
   • Again, VDE is redundant here.

6. Same as 2) except the QEMU guests are on a Virtual Network on another physical machine which is, in turn, connected to the Virtual Network on the first physical machine

     +-----------+                   D           +-----------+   |   Guest   |           N   D   H           |   Guest   |   |     A     |           A   N   C           |     B     |   |   +---+   |           T   S   P           |   +---+   |   |   |NIC|   |           ^   ^   ^           |   |NIC|   |   +---+-+-+---+           +---+---+           +---+-+-+---+         ^                     |                     ^               |   +--------+    +---+---+    +--------+   |               +-->+ vif1.0 +----+ vnbr0 +----+ vif2.0 +<--+                   +--------+    +---+---+    +--------+                                         |                                                       +---+---+                          | vtap0 |                          +---+---+                              |                           +--+--+                           | VDE |                           +--+--+                              |   First Physical Machine     V   -------------------------------------------------------------   Second Physical Machine    ^                              |              +-------+    +--+--+   +-------+        +---->+ VLAN0 +----+ VDE +---+ VLAN0 +<-----+        |     +-------+    +-----+   +-------+      |        V                                           V   +---+-+-+---+                               +---+-+-+---+  |   |NIC|   |                               |   |NIC|   |  |   +---+   |                               |   +---+   |  |   Guest   |                               |   Guest   |  |     C     |                               |     D     |  +-----------+                               +-----------+

   Notes:

   • What's going on here is that the two VDEs are connected over the network, either via a plan socket or perhaps encapsulated in another protocol like SSH or TLS

One interesting thing to note from all of those examples is thatalthough QEMU'snetworking options are very interesting, it doesn't actually makesense for a network to be implemented inside a guest. The networkneeds to be external to any guests, and so we use VDE to offer similarnetworking options to the ones QEMU provides. All QEMU needs to beable to do is to connect to VDE.

User Interface

This isn't meant a UI specification, but just some notes on how thisstuff might be exposed in virt-manager.

• Networks List:
  • Name
  • Virtual/Physical
  • Status
  • Activity/traffic
• Virtual Network Configuration:
  • Name
  • List of connected guests
  • Allow other Virtual Networks to connect to this (defaults to no)
  • Connect to other Virtual Network (defaults to none)
  • DHCP enabled - DHCP configuration:
    • IP range (optional)
    • Router IP address (optional)
    • Guest IP address/hostname assignment (optional)
  • Forwarding enabled - firewall configuration:
    • Incoming ports list and destination guest+port for each (defaults to empty)
    • Blocked outgoing ports lists (defaults to empty)
• Virtual NICs list:
  • Guest interface name
  • Virtual Network/Shared Physical Interface
  • Hostname (defaults to guest name)
  • IP address (if assigned)
  • MAC address (if assigned)
• Virtual NIC Configuration:
  • Random MAC address, or user-supplied MAC address.
  • Virtual Network or Shared Physical Interface to connect to.

Implementation

Parity with the current state of networking with Xen will be achievedby:

• Implementing "shared physical interface" support in Fedora's initscripts and network configuration tool. It boils down to configuring the interface (e.g. eth0) something like:

       ifcfg-peth0:       DEVICE=peth0       ONBOOT=yes       Bridge=eth0       HWADDR=00:30:48:30:73:19     ifcfg-eth0       DEVICE=eth0       Type=Bridge       ONBOOT=yes       BOOTPROTO=dhcp   

• Fixing Xen so that netloop is no longer required. Upstream have ideas about how to make Xen automatically copy any frames that are destined for Dom0 so that the netback driver doesn't run out of shared pages if Dom0 doesn't process the frames quickly enough.
• Create new network/vif scripts for Xen which will connect guests to a shared physical interface's bridge.

Virtual Networks will be implemented in libvirt. First, there will bean XML description of Virtual Networks e.g.:

  <network id="0">    <name>Foo</name>    <uuid>596a5d2171f48fb2e068e2386a5c413e</uuid>    <listen address="172.31.0.5" port="1234" />    <connections>      <connection address="172.31.0.6" port="4321" />    </conections>    <dhcp enabled="true">      <ip address="10.0.0.1"           netmask="255.255.255.0"           start="10.0.0.128"          end="10.0.0.254" />    </dhcp>    <forwarding enabled="true">      <incoming default="deny">        <allow port="123" domain="foobar" destport="321" />      </incoming>      <outgoing default="allow">        <deny port="25" />      </outgoing>    </forwarding>  <network>

In a manner similar to libvirt's QEMU support, there will be a daemonto manage Virtual Networks. The daemon will have access to a store ofnetwork definitions. The daemon will be responsible for managing thebridge devices, vde_switch/dhcp/dnses processes and the iptables rulesneeded for SNAT/DNAT etc.

virsh command line interface would look like:

  $> virsh network-create foo.xml  $> virsh network-dumpxml > foo.xml  $> virsh network-define foo.xml  $> virsh network-list  $> virsh network-start Foo  $> virsh network-stop Foo  $> virsh network-restart Foo

The libvirt API for virtual networks would be modelled on the API forvirtual machines:

/* * Virtual Networks API *//** * virNetwork: * * a virNetwork is a private structure representing a virtual network. */typedef struct _virNetwork virNetwork;/** * virNetworkPtr: * * a virNetworkPtr is pointer to a virNetwork private structure, this is the * type used to reference a virtual network in the API. */typedef virNetwork *virNetworkPtr;/** * virNetworkCreateFlags: * * Flags OR'ed together to provide specific behaviour when creating a * Network. */typedef enum {    VIR_NETWORK_NONE = 0} virNetworkCreateFlags;/* * List active networks */intvirConnectNumOfNetworks(virConnectPtr conn);intvirConnectListNetworks(virConnectPtr conn, int *ids, int maxids);/* * List inactive networks */intvirConnectNumOfDefinedNetworks(virConnectPtr conn);intvirConnectListDefinedNetworks(virConnectPtr conn, const char **names, int maxnames);/* * Lookup network by name, id or uuid */virNetworkPtrvirNetworkLookupByName(virConnectPtr conn, const char *name);virNetworkPtr virNetworkLookupByID(virConnectPtr conn, int id);virNetworkPtr virNetworkLookupByUUID(virConnectPtr conn, const unsigned char *uuid);virNetworkPtrvirNetworkLookupByUUIDString(virConnectPtr conn, const char *uuid);/* * Create active transient network */virNetworkPtrvirNetworkCreateXML(virConnectPtr conn, const char *xmlDesc, unsigned int flags);/* * Define inactive persistent network */virNetworkPtrvirNetworkDefineXML(virConnectPtr conn, const char *xmlDesc);/* * Delete persistent network */intvirNetworkUndefine(virNetworkPtr network);/* * Activate persistent network */intvirNetworkCreate(virNetworkPtr network);/* * Network destroy/free */intvirNetworkDestroy(virNetworkPtr network);intvirNetworkFree(virNetworkPtr network);/* * Network informations */const char*virNetworkGetName(virNetworkPtr network);unsigned intvirNetworkGetID(virNetworkPtr network);intvirNetworkGetUUID(virNetworkPtr network, unsigned char *uuid);intvirNetworkGetUUIDString(virNetworkPtr network, char *buf);char *virNetworkGetXMLDesc(virNetworkPtr network, int flags);

Discussion points on the XML format and API:

• The XML format isn't thought out at all, but briefly:
  • The <listen> and <connections> elements describe networks connected across physical machine boundaries.
  • The <dhcp> element describes the configuration of the DHCP server on the network.
  • The <forwarding> element describes how incoming and outgoing connections are forwarded.
• Since virConnect is supposed to be a connection to a specific hypervisor, does it make sense to create networks (which should be hypervisor agnostic) throughvirConnect?
• Are we needlessly replicating any mistakes from the domains API here? e.g. is the transient vs. persistent distinction useful for networks?
• Is a UUID useful for networks? Yes, because it distinguishes between networks of the same name on different hosts?
• Where is the connection between domains and networks in either the API or the XML format? How is a domain associated with a network? You put a bridge name in the <network>l definition and use that in the domains <interface> definition? Or you put the network name in the interface definition and have libvirt look up the bridge name when creating the guest?
• Should it be possible to stop/start/restart a network? What for? If something breaks the user restarts it to see if that will fix it?

Mark McLoughlin.Jan 15, 2007.