http://jason.digitalinertia.net/dockered-dpdk-packaging-open-vswitch/

Dockered DPDK: packaging Open vSwitch

FacebookTwitterGoogle+LinkedIn分享

I recently attended the NFV World Congress in San Jose, and had a great time talking to vendors about their solutions and current trends toward widespread NFV adoption. Intel’s hot new(ish) multicore programming framework – the Data Plane Development Kit, or DPDK – was part of the marketing spiel of almost everyone even remotely invested in the NFVI.  The main interest is in the poll mode driver, which dedicates a CPU core to polling devices rather than waiting for interrupts to signal when a packet has arrived.  This has resulted in some amazing packet processing rates, such as a DPDK-accelerated Open vSwitch switching at 14.88Mp/s.

Since I’ve been working with Docker lately, I naturally started imagining what could be done with combining crazy fast DPDK applications with the lightweight virtualization and deployment flexibility of Docker.  Many DPDK applications – such as Open vSwitch – have some requirements in the DPDK build that may break other applications if they relied on the same libraries.  This makes it a great candidate for containerization, since we can give the application its very own tested build and run environment.

I was not, of course, the first to think of this – some Googling will turn up quite a few bits and pieces that have been helpful in writing this post.  My goal here is to bring that information into a consolidated tutorial and to explain the containerized DPDK framework that I have published to Dockerhub.

DPDK Framework in a Container

DPDK applications need to access a set of headers and libraries for compilation, so I decided to create a base container (Github,Dockerhub) with those resources.  Here’s the Dockerfile:

FROM ubuntu:vividRUN apt-get update && apt-get install -y --no-install-recommends \ gcc build-essential make curl \ && apt-get clean && rm -rf /var/lib/apt/lists/*ENV DPDK_VERSION=2.0.0 \ RTE_SDK=/usr/src/dpdkRUN curl -sSL http://dpdk.org/browse/dpdk/snapshot/dpdk-${DPDK_VERSION}.tar.gz | tar -xz; \ mv dpdk-${DPDK_VERSION} ${RTE_SDK}# don't build kernel modulesRUN sed -i s/CONFIG_RTE_EAL_IGB_UIO=y/CONFIG_RTE_EAL_IGB_UIO=n/ ${RTE_SDK}/config/common_linuxapp \ && sed -i s/CONFIG_RTE_LIBRTE_KNI=y/CONFIG_RTE_LIBRTE_KNI=n/ ${RTE_SDK}/config/common_linuxapp# don't build unnecessary stuff, can be reversed in dpdk_config.shRUN sed -i s/CONFIG_RTE_APP_TEST=y/CONFIG_RTE_APP_TEST=n/ ${RTE_SDK}/config/common_linuxapp \ && sed -i s/CONFIG_RTE_TEST_PMD=y/CONFIG_RTE_TEST_PMD=n/ ${RTE_SDK}/config/common_linuxapp# set RTE_TARGET by sourcing dpdk_env.sh, should be in the build dir of the child image# must be sourced in beginning of any RUN instruction that needs itONBUILD COPY dpdk_env.sh ${RTE_SDK}/ONBUILD COPY dpdk_config.sh ${RTE_SDK}/ONBUILD RUN \ . ${RTE_SDK}/dpdk_env.sh; \ . ${RTE_SDK}/dpdk_config.sh; \ cd ${RTE_SDK} \ && make install T=${RTE_TARGET} \ && make clean

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

FROMubuntu:vivid

 

RUNapt-getupdate&&apt-getinstall-y--no-install-recommends\

  gcc build-essentialmakecurl\

  &&apt-getclean&&rm-rf/var/lib/apt/lists/*

 

ENVDPDK_VERSION=2.0.0\

  RTE_SDK=/usr/src/dpdk

 

RUN curl-sSLhttp://dpdk.org/browse/dpdk/snapshot/dpdk-${DPDK_VERSION}.tar.gz|tar-xz;\

  mvdpdk-${DPDK_VERSION}${RTE_SDK}

 

# don't build kernel modules

RUN sed-is/CONFIG_RTE_EAL_IGB_UIO=y/CONFIG_RTE_EAL_IGB_UIO=n/${RTE_SDK}/config/common_linuxapp\

  &&sed-is/CONFIG_RTE_LIBRTE_KNI=y/CONFIG_RTE_LIBRTE_KNI=n/${RTE_SDK}/config/common_linuxapp

 

# don't build unnecessary stuff, can be reversed in dpdk_config.sh

RUN sed-is/CONFIG_RTE_APP_TEST=y/CONFIG_RTE_APP_TEST=n/${RTE_SDK}/config/common_linuxapp\

  &&sed-is/CONFIG_RTE_TEST_PMD=y/CONFIG_RTE_TEST_PMD=n/${RTE_SDK}/config/common_linuxapp

 

# set RTE_TARGET by sourcing dpdk_env.sh, should be in the build dir of the child image

# must be sourced in beginning of any RUN instruction that needs it

ONBUILDCOPYdpdk_env.sh${RTE_SDK}/

ONBUILD COPY dpdk_config.sh${RTE_SDK}/

 

ONBUILD RUN\

  .${RTE_SDK}/dpdk_env.sh;\

  .${RTE_SDK}/dpdk_config.sh;\

  cd${RTE_SDK}\

  &&makeinstallT=${RTE_TARGET}\

  &&makeclean

Pretty basic stuff at first – get some packages, set the all-important RTE_SDK environment variable, grab the source.  One thing that is important is to not rely on kernel headers; doing so would be seriously non portable.  Theuio andigb_uio kernel modules have to be built and installed by the host that will run the DPDK container. Therefore, we configure the SDK to not compile kernel modules, and therefore not require installing kernel headers on the build system.

The key part of this build script is the deferment of compilation to when the application is built, so that the application can specify its requirements. This is done by requiring the DPDK application providedpdk_env.sh anddpdk_config.sh, which provide environment variables (such asRTE_TARGET) and a set of commands to run before compilation occurs. For example, Open vSwitch requires that DPDK be compiled withCONFIG_RTE_BUILD_COMBINE_LIBS=y in its configuration, which would be inserted indpdk_config.sh.

DPDK Application in a Container

Now that the framework is there, time to use it in an application.  In this post I will demonstrate Open vSwitch in a container (Github,Dockerhub), which could be plenty useful.  To begin, here’s the dpdk_env.sh and dpdk_config.sh files:

#!/bin/bashexport RTE_TARGET=x86_64-native-linuxapp-gcc

1

2

#!/bin/bash

export RTE_TARGET=x86_64-native-linuxapp-gcc

#!/bin/bash# OVS needs vhost library in DPDK, requires libfuse-dev to compileapt-get update && apt-get install -y --no-install-recommends \ libfuse-dev \ && apt-get clean && rm -rf /var/lib/apt/lists/*sed -i s/CONFIG_RTE_BUILD_COMBINE_LIBS=n/CONFIG_RTE_BUILD_COMBINE_LIBS=y/ $RTE_SDK/config/common_linuxappsed -i s/CONFIG_RTE_LIBRTE_VHOST=n/CONFIG_RTE_LIBRTE_VHOST=y/ $RTE_SDK/config/common_linuxappsed -i s/CONFIG_RTE_LIBRTE_VHOST_USER=y/CONFIG_RTE_LIBRTE_VHOST_USER=n/ $RTE_SDK/config/common_linuxapp

1

2

3

4

5

6

7

8

9

10

#!/bin/bash

 

# OVS needs vhost library in DPDK, requires libfuse-dev to compile

apt-getupdate&&apt-getinstall-y--no-install-recommends\

  libfuse-dev\

  &&apt-getclean&&rm-rf/var/lib/apt/lists/*

 

sed -is/CONFIG_RTE_BUILD_COMBINE_LIBS=n/CONFIG_RTE_BUILD_COMBINE_LIBS=y/$RTE_SDK/config/common_linuxapp

sed-is/CONFIG_RTE_LIBRTE_VHOST=n/CONFIG_RTE_LIBRTE_VHOST=y/$RTE_SDK/config/common_linuxapp

sed -is/CONFIG_RTE_LIBRTE_VHOST_USER=y/CONFIG_RTE_LIBRTE_VHOST_USER=n/$RTE_SDK/config/common_linuxapp

OVS has some special requirements for DPDK, which is kind of the point of putting it in a container, right? Here’s the Dockerfile to build it:

FROM rakurai/dpdk:2.0.0-onbuildRUN apt-get update && apt-get install -y --no-install-recommends \ autoconf automake libtool openssl libssl-dev python \ && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*ENV OVS_DIR=/usr/src/ovsRUN curl -ksSL http://github.com/openvswitch/ovs/archive/master.tar.gz | tar -xz; \ mv ovs-master ${OVS_DIR}RUN . ${RTE_SDK}/dpdk_env.sh; \ cd ${OVS_DIR} \ && ./boot.sh \ && ./configure --with-dpdk=${RTE_SDK}/${RTE_TARGET} \ && make install CFLAGS='-O3 -march=native' \ && make clean# create database configurationRUN ovsdb-tool create /usr/local/etc/openvswitch/conf.db /usr/local/share/openvswitch/vswitch.ovsschemaCOPY run_ovs.sh run_ovs.shRUN chmod +x run_ovs.shCMD ["./run_ovs.sh"]

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

FROMrakurai/dpdk:2.0.0-onbuild

 

RUNapt-getupdate&&apt-getinstall-y--no-install-recommends\

  autoconf automakelibtoolopenssllibssl-devpython\

  &&apt-getclean&&rm-rf/var/lib/apt/lists/*/tmp/*/var/tmp/*

 

ENVOVS_DIR=/usr/src/ovs

 

RUNcurl-ksSLhttp://github.com/openvswitch/ovs/archive/master.tar.gz|tar-xz;\

  mvovs-master${OVS_DIR}

 

RUN .${RTE_SDK}/dpdk_env.sh;\

  cd${OVS_DIR}\

  &&./boot.sh\

  &&./configure--with-dpdk=${RTE_SDK}/${RTE_TARGET}\

  &&makeinstallCFLAGS='-O3 -march=native'\

  &&makeclean

 

# create database configuration

RUN ovsdb-toolcreate/usr/local/etc/openvswitch/conf.db/usr/local/share/openvswitch/vswitch.ovsschema

 

COPY run_ovs.shrun_ovs.sh

RUNchmod+xrun_ovs.sh

CMD ["./run_ovs.sh"]

The ONBUILD instructions in the DPDK Dockerfile will be executed first, of course, which will compile the DPDK framework. Then we install more packages for OVS, get the source, and compile with DPDK options. In the last few lines, we move the final script into the container, which is all the stuff OVS needs running:

#!/bin/bashexport DB_SOCK=/usr/local/var/run/openvswitch/db.sockovsdb-server --remote=punix:$DB_SOCK \ --remote=db:Open_vSwitch,Open_vSwitch,manager_options \ --private-key=db:Open_vSwitch,SSL,private_key \ --certificate=Open_vSwitch,SSL,certificate \ --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert \ --pidfile \ --detachovs-vsctl --no-wait initovs-vswitchd --dpdk -c 0x1 -n 4 -- unix:$DB_SOCK --pidfile

1

2

3

4

5

6

7

8

9

10

11

12

13

#!/bin/bash

export DB_SOCK=/usr/local/var/run/openvswitch/db.sock

 

ovsdb-server--remote=punix:$DB_SOCK\

  --remote=db:Open_vSwitch,Open_vSwitch,manager_options\

  --private-key=db:Open_vSwitch,SSL,private_key\

  --certificate=Open_vSwitch,SSL,certificate\

  --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert\

  --pidfile\

  --detach

 

ovs-vsctl--no-waitinit

ovs-vswitchd--dpdk-c0x1-n4--unix:$DB_SOCK--pidfile

Now, here you could go a bit differently, and the repository I linked to may change somewhat. It could be said that it is more Dockerish to put theovsdb-server in its own container, and then link them. However, this is a self contained example, so we’ll just go with this.

Running Open vSwitch

Before we start it up, we need to fulfill some prerequisites. I won’t go into details on the how and why, but please see theDPDK Getting Started Guide and theOVS-DPDK installation guide.  OVS requires 1GB huge pages, so you need your /etc/default/grub to have at least these options:

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1GB hugepagesz=1G hugepages=1"

1

GRUB_CMDLINE_LINUX_DEFAULT="default_hugepagesz=1GB hugepagesz=1G hugepages=1"

followed by an update-grub and reboot. You also need to mount them with this or the/etc/fstab equivalent:

mkdir -p /mnt/hugemount -t hugetlbfs -o pagesize=1G nodev /mnt/huge

1

2

mkdir-p/mnt/huge

mount -thugetlbfs-opagesize=1Gnodev/mnt/huge

Compile the kernel module on the host system and insert it. Download DPDK, extract, and run thedpdk/tools/setup.sh script. Choose to build to the x86_64-native-linuxapp-gcc target, currently option 9, and then insert the UIO module, currently option 12. Finally, bind one of your interfaces with option 18, though you’ll have to bring that interface down first.

Now you can start the container. Here’s what I used:

docker run --privileged -v /mnt/huge:/mnt/huge --device=/dev/uio0:/dev/uio0 rakurai/ovs-dpdk

1

dockerrun--privileged-v/mnt/huge:/mnt/huge--device=/dev/uio0:/dev/uio0rakurai/ovs-dpdk

This gives the container access to the huge page mount, and the uio0 device that you just bound to the UIO driver. I found that I needed to run the container as--privileged to access parts of the/dev/uio0 filesystem, though it appears that some people are able to get around this. I will update this post if I find out how to run the container without privileged.

If all goes well, you now have DPDK-accelerated OVS running in a container, and you can go about adding interfaces to the container, adding them to OVS, and setting up rules for forwarding packets at ludicrous speeds. Good luck, and please let me know how it works out for you!

Links

DPDK base Docker container – rakurai/dpdk – Github, Dockerhub
Open vSwitch Docker container – rakurai/ovs-dpdk – Github, Dockerhub
DPDK Getting Started Guide
OVS-DPDK installation guide

3 comments

1. Murad Kablan
   December 5, 2015 at 12:07 am

   Hey Jason,
   Thanks for this great article.
   I am having a problem running multiple containers with DPDK on the same host as the same time. It seems that they block each other when it comes to use of hugepages.
   This is the error I get when I try to run a DPDK application in the second container.
   EAL: Detected lcore 0 as core 0 on socket 0
   EAL: Detected lcore 1 as core 0 on socket 1
   EAL: Detected lcore 2 as core 1 on socket 0
   EAL: Detected lcore 3 as core 1 on socket 1
   EAL: Detected lcore 4 as core 2 on socket 0
   EAL: Detected lcore 5 as core 2 on socket 1
   EAL: Detected lcore 6 as core 3 on socket 0
   EAL: Detected lcore 7 as core 3 on socket 1
   EAL: Detected lcore 8 as core 4 on socket 0
   EAL: Detected lcore 9 as core 4 on socket 1
   EAL: Detected lcore 10 as core 5 on socket 0
   EAL: Detected lcore 11 as core 5 on socket 1
   EAL: Support maximum 128 logical core(s) by configuration.
   EAL: Detected 12 lcore(s)
   EAL: No free hugepages reported in hugepages-1048576kB
   PANIC in rte_eal_init():

   Any advice on how to solve this issue?

   Thanks,
   Murad

   Reply
   1. admin
      December 6, 2015 at 6:55 pm

      Hi Murad, thanks for reading. I haven’t attempted running more than one application at once, but if you’re using something like the Dockerfile I posted, the last line of run_ovs.sh:

      ovs-vswitchd --dpdk -c 0x1 -n 4 -- unix:$DB_SOCK --pidfile

      might be the issue. Arguments before the ” — ” are DPDK args, such as “-c 0x1″ being the core mask to tell it to only run on core 0. As noted in the OVS documentation for DPDK installation, you can specify memory restrictions if you have more huge pages allocated with an argument like

      --socket-mem 1024,0

      to allocate 1GB on NUMA node 0. So changing the last line of the script to

      ovs-vswitchd --dpdk -c 0x1 -n 4 --socket-mem 1024,0 -- unix:$DB_SOCK --pidfile

      may solve your problem if 1 core and a single 1GB huge page suits your use case. Of course, doing something fancy with the command line args for the script and Dockerfile would be more elegant, but you get the idea.

      Let me know if this helps!

      Reply
      1. Murad Kablan
         December 8, 2015 at 12:51 am

         Thanks for the feedback.
         So I found out what was going on and the problem falls into two issues.
         First issue is the memory. As you said, I had to limit each container’s memory usage. But I also had to specify that each container is working independently from other containers using the option –file-prefix

         The second issue is that DPDK applications block all PCI devices by default. So I when I ran the second container, the first container’s port was blocked. So I had to specify or “whiteliste” the port used by each container
         by using the option
         –pci-whitelist PCI-ID

         I’m about to run as much containers as possible on a single host to see if DPDK performance can be effected or not.

         Thanks again and please keep posting these cool articles!

         Reply

Leave a Reply