How to test PCIe SR-IOV in D05

-v0.1 2017.2.9 Sherlock init
-v0.2 2017.2.19 Sherlock add pcie_acs_override reference

This doc helps to introduce how to test PCIe SR-IOV in HiSilicon D05 board.

0. preparation

kernel: https://github.com/hisilicon/kernel-dev.git branch: private-topic-sriov-v3-4.10

UEFI: openlab1.0 101 server /home/wangzhou/repo/plinth_uefi/uefi

qemu: https://github.com/eauger/qemu.git branch: v2.7.0-passthrough-rfc-v5

hardware topology:

   +---------------+       +----------------------+   |  D05 I        |       |         D05 II       |   |               |       |                      |   |   +-----------+       +------------------+   |   |   |1P NA PCIe2|<----->|any 10G networking|   |   |   +-----------+       +------------------+   |   +---------------+       +----------------------+

1. compile kernel and UEFI

configure kernel: Add SMMU_V3=y, 82599 PF driver = m, 82599 VF driver = m,
VFIO PCI driver = m (p.s. ACPI boot)
compile kernel image and ko
compile UEFI using: ./uefi-tools/uefi-build.sh -c ./LinaroPkg/platforms.config d05

2. basic test

1. boot up host OS (firstly update UEFI above)[1]2. copy modules to host OS:   ixgbe.ko ixgbevf.ko mdio.ko vfio_iommu_type1.ko vfio.ko vfio-pci.ko   vfio_virqfd.ko irqbypass.ko3. prepare host environment:   mkdir /lib/modules/`uname -r`   touch /lib/modules/`uname -r`/modules.order   touch /lib/modules/`uname -r`/modules.builtin   depmod ixgbe.ko ixgbevf.ko mdio.ko vfio_iommu_type1.ko vfio.ko vfio-pci.ko          vfio_virqfd.ko irqbypass.ko   modprobe ixgbe   modprobe ixgbevf   /* trigger one VF, 0002:81:00.0 is the PF in which you want trigger a VF */   echo 1 > /sys/devices/pci0002:80/0002:80:00.0/0002:81:00.0/sriov_numvfs   modprobe -v vfio-pci disable_idle_d3=1   modprobe -r vfio_iommu_type1   modprobe -v vfio_iommu_type1 allow_unsafe_interrupts=1   /* set related PF up */   ifconfig eth26 up   /* 0002:81:10.0 is BDF of VF */   echo vfio-pci > /sys/bus/pci/devices/0002:81:10.0/driver_override   echo 0002:81:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind   echo 0002:81:10.0 > /sys/bus/pci/drivers_probe4. run qemu[2]   qemu-system-aarch64 \   -machine virt,gic-version=3 \   -enable-kvm \   -cpu host \   -m 1024 \   -kernel ./Image \   -initrd ./minifs.cpio.gz \   -nographic \   -net none -device vfio-pci,host=0002:81:10.0,id=net0 \   -D trace_log5. set networking configurations in guest machine and remote machine   run ping and iperf to test

3. more scenarios

1. enable multiple VFs, assigned to one VM2. enable multiple VFs, assigned to different VMs3. use VF directly4. VF and PF communicate

4. performance

1. should at least reach the performance of PF[3]

reference:
[1] should add pcie_acs_override=downstream in kernel command line
[2] how to compile qemu locally in D05:
http://blog.csdn.net/scarecrow_byr/article/details/51494020
[3] should add 82599 patch to test, firstly make sure performance of 82599 PF is
good.