Vitual Router in The Cloud

VyOS and ESXi,VyOS Configuration

The next step is to configure both VyOS routers. Before we do, we should ensure that we have a good high-level understanding of what should be happening.

The ultimate goal of this three-router setup is to have our own VyOS router as the gateway to the Internet, while also allowing the Verizon router to continue providing network access for the value-added services like:

• Video-on-Demand to set-top boxes
• On-screen caller ID
• Remote DVR access
• Etc.

The Verizon router does this by setting up its own NAT’d network on the 192.168.1.0/24 range, which the STBs in the house sit on and use to communicate with Verizon’s servers. The VZ router expects and requires the IP it is assigned on its WAN port to be publicly routable on the FiOS ISP network. If it is not, things may or may not work, or they might become unpredictable in their functionality.

The entire point of the secondary router is to provide 1:1 NAT between the home network and the VZ router, so that the VZ router gets assigned the same IP as the primary router that is actually talking to the FiOS ISP network.

With three different Layer 2 domains and some creative port forwarding, the Verizon router won’t even know the difference.

This network configuration, combined with some port forwarding rules on the primary and secondary router (discussed later), allows traffic between the Verizon router and the Verizon servers to flow normally without the VZ router being aware that it is not actually directly connected to the FiOS ISP network.

Let’s start by configuring the primary router. This router will actually receive the public-facing IP from the FiOS ISP network, and thus will ultimately be responsible for all Internet traffic. Log into your primary router and run the show interfaces command.

vyos@primary-router:~$ show interfacesCodes: S - State, L - Link, u - Up, D - Down, A - Admin DownInterface        IP Address                        S/L  Description---------        ----------                        ---  -----------eth0             108.0.0.123/24                    u/u  FiOS Public Internet eth1             10.0.0.1/24                       u/u  Home Network lo               127.0.0.1/8                       u/u                   ::1/128

We see two Ethernet interfaces, eth0 and eth1. These represent the two vNICs provisioned to this VM, and which correspond to the FiOS Public Network and Home Network port groups, respectively.

Let’s configure the eth0 interface first.

vyos@primary-router:~$ configurevyos@primary-router:~# set interfaces ethernet eth0 address dhcpvyos@primary-router:~# set interfaces ethernet eth0 description FiOS_ISP_Netvyos@primary-router:~# set interfaces ethernet eth0 duplex autovyos@primary-router:~# set interfaces ethernet eth0 speed auto

This will set this interface up to use a dynamically assigned address (from Verizon), set a description to make it easy to remember what it connects to, and auto negotiate speed and duplex settings.

There is one more step required. We must configure this interface to impersonate our Verizon hardware router’s WAN interface by setting it to use the same MAC address (Verizon filters MACs that are not on its whitelist). You can find the WAN MAC you need to enter printed on the bottom of your Verizon router. Replace 0a:1b:2c:3d:4e:5f below as appropriate:

vyos@primary-router:~# set interfaces ethernet eth0 mac 0a:1b:2c:3d:4e:5f

Let’s take a look at the changes we are making.

vyos@primary-router:~# compare

When you are satisfied, commit the changes to the running configuration and save the running config to disk. If you commit but do not save, the changes will not persist after a reboot of the router.

vyos@primary-router:~# commitvyos@primary-router:~# save

Let’s take another look at the interface configuration now. We’re still in configuration mode (note the # symbol at the end of the command prompt), so we need to prepend run to the command we used before.

vyos@primary-router:~# run show interfaces

Hopefully, your eth0 interface has a public address assigned from the Verizon DHCP server. If not, check your connections and configurations.

Assuming all is well, you should now be able to ping addresses to confirm that you have connectivity out to the Internet.

vyos@primary-router:~# run ping 8.8.8.8PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.64 bytes from 8.8.8.8: icmp_req=1 ttl=251 time=21.1 ms64 bytes from 8.8.8.8: icmp_req=2 ttl=251 time=22.0 ms64 bytes from 8.8.8.8: icmp_req=3 ttl=251 time=20.9 ms64 bytes from 8.8.8.8: icmp_req=4 ttl=251 time=22.3 ms^C--- 8.8.8.8 ping statistics ---4 packets transmitted, 4 received, 0% packet loss, time 3004msrtt min/avg/max/mdev = 20.948/21.610/22.307/0.605 ms

If this works, then congratulations! The good news is that your first router is working. The bad news is that nothing else can use your Internet connection yet.

VyOS and OpenStack Configuration Drives

VyOS is an open source fork of the last open source release of Vayatta, which turned proprietary a few years ago. We are currently using VyOS at work to set up OSPF routers in an OpenStack environment, and will soon have to spawn a very large amount of these in a proof-of-concept deployment.

This describes how we add support for OpenStack’s Configuration Drive to VyOS.

VyOS has something of an unhealthy relationship with Debian Squeeze (it is currently incompatible with newer Debian releases), and requires a Debian Squeeze installation in order to create the VyOS ISO used for deployments.

Below we will patch a post-installation script and add our own (very simple) Python-script that parses the Configuration Drive information and complies with a very small subset of what features packages like cloud-init provide. Unfortunately, cloud-init is not available for Debian Squeeze, which is the whole reason we are doing this in the first place.

Steps:

1. Install Squeeze
2. Create Python script
3. Run setup-script
4. Import ISO into Glance
5. Spawn OpenStack instance
6. Verify that it works

Prepare a little Python script for parsing the OpenStack Configuration Drive metadata:

#!/usr/bin/pythonimport jsonimport shutilmeta_data_file = open('/config-drive/openstack/latest/meta_data.json')json_input = meta_data_file.read()try:  decoded = json.loads(json_input)  for file in decoded['files']:    print file['content_path'], file['path']    shutil.copy2('/config-drive/openstack' + file['content_path'], file['path'])except (ValueError, KeyError, TypeError):  print "JSON format or content error"

Save this as process-openstack-metadata.py. This will be baked into the ISO in the script below as/root/vyos-init.py.

Below is a script to generate a VyOS ISO with a few modifications. Most of it is straight from the VyOS wiki page How to build an ISO image. Read through it so you see what it does and save it as build-vyos-iso.sh, chmod it (chmod +x build-vyos-iso.sh) and run it.

#!/bin/bash -xeapt-get install debian-archive-keyringcat >> /etc/apt/sources.list <<EOFdeb http://backports.debian.org/debian-backports squeeze-backports mainEOFapt-get update# Get backported version of squashfsapt-get -t squeeze-backports install squashfs-toolsapt-get install git autoconf automake dpkg-dev live-helper syslinux genisoimagebranch=hydrogen # hydrogen = stable, helium = devif ! test -d build-isothen  git clone https://github.com/vyos/build-iso.git  cd build-iso  git branch $branch --track origin/$branch  git checkout $branchelse  cd build-isofiif ! test -d pkgs/vyatta-cfg-system/debianthen  git submodule update --init pkgs/vyatta-cfg-system  cd pkgs/vyatta-cfg-system/  git branch $branch --track origin/$branch  git checkout $branchelse  cd pkgs/vyatta-cfg-system/fi# Reset debian/vyatta-cfg-system.postinst.in so we can patch it againgit checkout debian/vyatta-cfg-system.postinst.in# Patch debian/vyatta-cfg-system.postinst.inpatch -p0 <<"HEREDOC"--- debian/vyatta-cfg-system.postinst.in    2015-01-17 15:09:53.000000000 +0100+++ debian/vyatta-cfg-system.postinst.in.patched    2015-01-17 15:11:19.000000000 +0100@@ -143,6 +143,19 @@ # configuration is fully applied. Any modifications done to work around # unfixed bugs and implement enhancements which are not complete in the Vyatta # system can be placed here.++mkdir /config-drive++mount -o ro -t iso9660 /dev/disk/by-label/config-2 /config-drive++/root/vyos-init.py++configure+load /root/configuration+commit+save++umount /config-drive EOF fiHEREDOCcd -mkdir livecd/config.vyatta/chroot_local-includes/rootcp ../process-openstack-metadata.py \  livecd/config.vyatta/chroot_local-includes/root/vyos-init.pychmod +x livecd/config.vyatta/chroot_local-includes/root/vyos-init.pyaptitude install pdebuild-crossmake vyatta-cfg-systemfind pkgs -name 'vyatta-cfg-system*.deb' || \  (echo "pkgs/vyatta-cfg-system*.deb not found, exiting..."; exit 1)echo python-simplejson >> \  livecd/config.vyatta/chroot_local-packageslists/vyatta-extra.listexport PATH=/sbin:/usr/sbin:$PATHautoreconf -i./configuremake isols -l livecd/binary.isoecho Done!

If everything went well you will have an ISO at ./build-iso/livecd/binary.iso.

Upload this file into OpenStack with Glance and name it “VyOS Router”:

glance image-create --name "VyOS Router" --is-public True \  --disk-format iso --container bare < ./build-iso/livecd/binary.iso

Create your own config.boot (or whatever else you want on the deployed machine):

cat > config.boot <<"EOF"interfaces {    ethernet eth0 {        address dhcp    }    loopback lo {    }}service {    ssh {        port 22    }}system {    login {        user vyos {            authentication {                plaintext-password "demo"            }            level admin        }    }}EOF

Spawn an instance with a predefined flavor and our new configuration file to be included on the configuration drive:

nova boot --config-drive true --image "VyOS Router" \  --flavor <flavor> --file /root/configuration=config.boot \  --meta essential=false --nic net-id=<net-id> vyos

Verify that it works by logging into VyOS and check if the running configuration is the one you expect. You can start by checking if the file /root/configuration exists and if it the content is what you intended.

How to debug:

vyos@vyos:~$ /usr/sbin/tcpdump -f "icmp" -i eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
18:10:13.234909 IP 10.168.11.102 > 8.8.8.8: ICMP echo request, id 40962, seq 49, length 64
18:10:13.261277 IP 8.8.8.8 > 10.168.11.102: ICMP echo reply, id 40962, seq 49, length 64
18:10:14.235045 IP 10.168.11.102 > 8.8.8.8: ICMP echo request, id 40962, seq 50, length 64
18:10:14.261379 IP 8.8.8.8 > 10.168.11.102: ICMP echo reply, id 40962, seq 50, length 64
18:10:15.235249 IP 10.168.11.102 > 8.8.8.8: ICMP echo request, id 40962, seq 51, length 64
18:10:15.261549 IP 8.8.8.8 > 10.168.11.102: ICMP echo reply, id 40962, seq 51, length 64
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel
vyos@vyos:~$ /usr/sbin/tcpdump -f "icmp" -i eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
18:10:18.235887 IP XXX.XXX.187.78 > 8.8.8.8: ICMP echo request, id 40962, seq 54, length 64
18:10:18.262249 IP 8.8.8.8 > XXX.XXX.187.78: ICMP echo reply, id 40962, seq 54, length 64
18:10:19.236110 IP XXX.XXX.187.78 > 8.8.8.8: ICMP echo request, id 40962, seq 55, length 64
18:10:19.262477 IP 8.8.8.8 > XXX.XXX.187.78: ICMP echo reply, id 40962, seq 55, length 64
18:10:20.236345 IP XXX.XXX.187.78 > 8.8.8.8: ICMP echo request, id 40962, seq 56, length 64
18:10:20.262652 IP 8.8.8.8 > XXX.XXX.187.78: ICMP echo reply, id 40962, seq 56, length 64
18:10:21.236527 IP XXX.XXX.187.78 > 8.8.8.8: ICMP echo request, id 40962, seq 57, length 64
18:10:21.262927 IP 8.8.8.8 > XXX.XXX.187.78: ICMP echo reply, id 40962, seq 57, length 64
18:10:22.237082 IP XXX.XXX.187.78 > 8.8.8.8: ICMP echo request, id 40962, seq 58, length 64
18:10:22.263398 IP 8.8.8.8 > XXX.XXX.187.78: ICMP echo reply, id 40962, seq 58, length 64
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel
vyos@vyos:~$ ping 10.168.11.102
PING 10.168.11.102 (10.168.11.102) 56(84) bytes of data.
64 bytes from 10.168.11.102: icmp_req=1 ttl=64 time=0.481 ms
64 bytes from 10.168.11.102: icmp_req=2 ttl=64 time=0.559 ms